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Overview 


With the COVID-19 pandemic disrupting 
education across the globe in early 2020, 
many educators found themselves rapidly 
transitioning to remote learning without the 
appropriate knowledge, skills or resources. 
For many higher education students, this 
resulted in less than optimal learning 
experiences. 


To ensure that high quality online learning is being 
offered in the future, the Kingdom of Saudi Arabia 
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learning pre-, peri- and post-COVID. The intention was to 
determine opportunities for improvement while 
identifying areas of excellence that could be highlighted 
against an evaluation framework for quality online 
learning. Using this information, fall higher education 
programs within the Kingdom of Saudi Arabia would be 
positioned to make improvements to the overall student 
learning experience. 
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Executive Summary 


To ensure that high quality online learning is being offered in the future, the Kingdom 
of Saudi Arabia commissioned a study to understand the state of online learning pre-, 
peri- and post-COVID. The study culminated in a number of recommendations to allow 
the Kingdom of Saudi Arabia to effectively build infrastructure and capacity and to 
deliver high quality online learning 


The study focused on eight dimensions, each with several subdimensions, and included survey and interview 
data, contextualized within the KSA higher education environment for both development and analysis. 
Dimensions included Leadership, Curriculum Design and Planning, Online Teaching and Learning, Assessment, 
Technology, Student Support, Training and Support, and Evaluation and Continuous Improvement. Each of these 
dimensions and their associated subdimensions are discussed at length in the report, but there were several 
overarching themes that emerged. 


Themes 


Clear themes and trends emerged throughout the study. It is notable that there is a high degree of consistency in 
the responses of administrators/staff and faculty regarding their perceptions and experiences with online 
learning. Students also displayed consistency in their responses as a group, though there were discrepancies 
between students and administrators/staff and faculty. Although there were many themes throughout the study 
and certainly within dimensions, two themes recurred consistently, spanning all dimensions and multiple aspects 
of online learning. 


Communication emerged as a strong trend throughout each of the dimensions, and spanned all stakeholder 
groups. Key findings related to communication include: 

e Regular and clear communication between administrators/staff and faculty is necessary to ensure 
understanding of governance, strategy, and policies, as well as practices regarding online teaching and 
learning pedagogy and technology. Additionally, communication regarding needs spanning technology, 
training and support, and resources is critical to ensure effective implementation on online learning. 

e = In addition to communicating about these items, faculty continuously expressed a clear desire for more 
involvement in decision-making for online teaching and learning at all levels, including administration 
(strategy, policies) and development and delivery of online learning (curriculum, evaluation, etc.). 

e Frequent and effective communication between faculty and students emerged as a need. This is 
particularly evident in online teaching and learning, assessment, and student support dimensions. 
Communication of expectations (performance, interaction, etc.), the navigation of online courses, 
assessment practices and purposes, and expectations regarding communication itself (feedback, 
interaction, etc.) is necessary to provide equitable opportunities for student success. 


Resource allocation and the availability of resources was a common trend throughout the study. Key findings 
related to resources include: 

e Technology resources, including both infrastructure and the availability of devices, was a trend 
throughout the study. Adequate technology resources, including the ability of the infrastructure to 
support high-volume use as well as to provide centralized access to online learning was a need that was 
repeatedly identified. Additionally, ensuring that both students and faculty have access to devices that 
support the online learning environment and are available when needed was a frequent theme in survey 
and interview responses. 

e Internet access was a critical issue that recurred throughout the study. Access to the internet overall, as 
well as the ability of the available network to adequately handle the demands of online teaching and 
learning emerged as a recurrent theme, including access at both school and home locations. 

e Related to technology, consistent needs were identified related to online teaching and support, including 
faculty training and support for both online technologies and pedagogies and support for online course 
design and delivery as well as student training and support on technology. Increased support in these 
areas will not only help to ensure smooth technological experiences, but result in higher quality online 
courses to support successful student learning outcomes. 


Emerging Needs 


Survey and interview analyses resulted in recommendations across all eight dimensions that will allow for the 
development and implementation of higher capacity and quality in online learning. Themes throughout, and 
particularly related to communication and technology indicate that though some challenges existed prior to 
COVID-19, the rapid shift to online learning following the pandemic resulted in increased visibility of challenges 
(both prior to and following COVID-19) as well as increased need in these areas. Study results indicate that 
immediate action was taken on these items, though challenges remain. Moving forward, direct attention to 
technology and communication, particularly between administrators/staff and faculty, and between faculty and 
students, is essential to the development and delivery of effective online teaching and learning opportunities. 


Scenarios 


The two scenarios presented from this data include online teaching and learning goals outside of consideration 
for COVID, and implementing effective online teaching and learning as influenced by COVID. 


Recommendations provided encompass both scenarios, although some present long-term strategies for 
increasing the capacity and effectiveness of online teaching and learning (such as long-range strategic planning, 
development of processes for evaluation and continuous improvement, and implementation of policies regarding 
faculty input for curriculum and decision-making). Others, while no less important for long-term solutions, are a 
higher priority to manage effective learning experiences as well as support needs during COVID (for example, 
ensuring that technology is sufficient to support high-volume online courses and providing clear expectations for 
communication, assessment, and interactions in the online environment). All recommendations are discussed in 
detail in the report, and summarized (including identification of COVID priorities) in Appendix IV: Future Action 
Framework. 
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Implications 


Prior to COVID-19 and the commissioning of this study, the Kingdom of Saudi Arabia included online teaching and 
learning as a need and goal of Vision 2030, and invested in the development and success of online education as a 
key component of attaining long-term higher education goals. This study illuminated overall needs to enhance 
country-wide needs related to accomplishing these goals. The pandemic required rapid acceleration in the 
development and delivery of fully online courses for a high volume of students, and in addition to existing needs, 
this study further illuminated critical needs to support success. 


The recommendations resulting from this study are designed to support both long-term and immediate needs to 
develop and deliver effective online teaching and learning in both scenarios. While many elements necessary to 
support online education were evident pre-COVID, and clear efforts have been made to provide effective online 
learning during the pandemic, implications resulting from this study indicate that there are many actions that may 
be implemented to better support Vision 2030 goals and address needs that emerged with COVID-19 and persist 
throughout the shift resulting from the pandemic. 


Introduction 


With the COVID-19 pandemic disrupting education across the globe in early 2020, 
many educators found themselves rapidly transitioning to remote learning without 
the appropriate knowledge, skills or resources. For many higher education students, 
this resulted in less than optimal learning experiences. 


To ensure that high quality online learning is being offered in the future, the Kingdom of Saudi Arabia 
commissioned a study to understand the state of online learning pre-, peri- and post-COVID. The intention was to 
determine opportunities for improvement while identifying areas of excellence that could be highlighted against 
an evaluation framework for quality online learning. Using this information, fall higher education programs within 
the Kingdom of Saudi Arabia would be positioned to make improvements to the overall student learning 
experience. 


Given their vast network and expertise, the Online Learning Consortium (OLC) was asked to lead this project with 
the assistance of several key partners from around the world. This included the International Council for Open and 
Distance Education, UNESCO IITE, Quality Matters, the University of Wisconsin-Milwaukee Center for Distance 
Education and Technological Advancement (DETA), and the Association of Public and Land-Grant Universities 
(APLU). Partners were included at different stages of the study but primarily provided feedback and input into the 
analysis and recommendations provided within this report. 


This report has been organized into four primary sections and includes several appendices. The introduction 
provides context for the study, the methodology section provides information regarding the development and 
analysis of both quantitative survey instrumentation and qualitative protocols as well as a summary of 
participants, the recommendations section provides data interpretation and recommendations by dimension and 
subdimension (aligned with the Future Action Framework), and the considerations for future study section 
contains recommendations for further research based on this study as well as findings indicating potential future 
expansion. Finally, the appendices provide details on evaluation, research, and action frameworks, a glossary of 
common terminologies recommended resources, the literature review, survey and interview protocols, and a full 
listing of tables, charts, and graphs. 
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Methodology 


The higher education study began with the development of an eight-dimension 
framework, designed by NELC and collaboratively finalized by NELC and OLC. 


The framework, including dimensions and subdimensions, is attached in Appendix I: NELC Evaluation Framework, 
and was used to inform both survey and interview protocols for evaluation, as well as to develop Appendix II: 
Research Methodology Framework Mapping. Appendix IV: Future Action Framework, is the culmination of the 
evaluation and research frameworks with recommendations summarized as described later in this section. 


Drawing on the eight dimensions and subdimensions, and informed by policy documents provided by NELC, 
descriptive surveys were developed for higher education administrators/staff, faculty, and students by DETA and 
OLC. The purpose of the surveys was to understand the national position to provide quality online learning across 
institutions before and during the COVID-19 pandemic in the Kingdom of Saudi Arabia (KSA) and to identify areas 
for future investment and improvement. The recommendations resulting from survey analyses provide 
information to support capacity and quality in online education throughout the country to enhance the potential 
for Vision 2030 excellence in a diversified and knowledge-based economy. Survey instrumentation was developed 
based on ascan of other instrumentation and tools, including national and international products, literature, and 
recent research relevant to the dimensions and subdimensions. Survey items were developed to measure the 
participants’ attitudes, opinions, and beliefs about online learning at their institutions, to assess the performance 
of the institutions and different functions of the institutions to support online learning and quality, and to identify 
areas of challenge and opportunities for improvement in future academic terms. All survey items are considered 
to have both construct and content validity. 


The survey was delivered through Survey Monkey and was designed as a single instrument with branching logic to 
ease data collection and dataset management. Items were individually coded by stakeholder group and 
dimension for analysis and storage. Survey distribution was requested through university presidents, who 
provided the survey link to other administrators, staff, faculty and students. 


Survey analysis included descriptive statistical analysis completed in SPSS, disaggregated by stakeholder group 
(higher education administrators/staff, faculty, and students) as well as by dimension. 


There were 24,409 completed surveys, including 832 administrators (3.4%), 2,401 staff members (9.8%), 6,171 
faculty members (25.3%), and 15,005 students (61.5%). Please note that a number of respondents chose not to 
answer demographic questions beyond their role, and the figures described below represent those that did 
provide responses. 


= Administrator/staff participants: There were a total of 848 administrator/staff participants, with 616 
(72.6%) from public universities and 232 (27.4%) from private universities. There were more male 
respondents (517, 59.4%) than female (353, 40.6%), and the majority of the sample (455, 54.4%) was 
made up of staff members, with 30 (7.2%) presidents/vice presidents, 115 (13.7%) deans/vice deans, and 
237 (28.3%) with ‘other’ roles. 


= Faculty participants: A total of 2,522 faculty participated, including 2,093 (83.5%) from public 
universities and 413 (16.5%) from private universities. The majority of faculty participants identified as 
non-Saudi (1,685, 67.0%) with 829 (33.0%) indicating Saudi nationality. There were 1,509 (59.8%) male 
and 1,013 (40.2%) female participants. The sample included a variety of faculty roles: 245 (9.8%) 
professors, 415 (16.6%) associate professors, 1,187 (47.4%) assistant professors, 507 (20.2%) lecturers, 
101 (4.0%) teaching assistants, and 51 (0.8%) with other roles. Participants also represented a range of 
experience levels, with 411 (16.4%) with five years or less, 726 (28.9%) with more than five to ten years, 
463 (18.4%) with ten to fifteen years, and 912 (36.3%) with more than fifteen years of experience. 
Participants were from diverse fields of study, with 904 (14.6%) in Human Sciences, 488 (36.3%) in Natural 
Sciences and Engineering, 169 (6.8%) in Information and Computer Sciences, 563 (22.6%) in Medical and 
Health Sciences, and 363 (14.6%) in other fields. 


= Student participants: A total of 4,902 students participated in the survey. Public university students 
made up 76.0% (3,564) of the sample, with 24.0% (1,127) from private universities. Unlike 
administrators/staff and faculty, more female students (2,736, 55.8%) than male (2,166, 44.2%) 
participated. Student participants were primarily traditional college-aged, with 57.2% (2,806) between 
the ages of 18-22, followed by 28.6% (1,403) aged 23-27, 6.9% (339) aged 28-32, 3.3% (164) aged 32-36, 
and 4.0% (197) over age 36. The student sample included 204 (4.4%) seeking a diploma, 4,102 (87.7%) 
seeking a bachelor’s degree, 22 (0.5%) seeking a higher diploma, 6.5% (305) seeking a master’s degree, 
and 1.0% (46) seeking a doctoral degree. Diverse fields were represented in the student sample, with 711 
(15.4%) in Human Sciences, 661 (14.3%) in Natural Sciences and Engineering, 497 (10.8%) in Information 
and Computer Sciences, 894 (19.4%) in Medical and Health Sciences, and 1,845 (40.0%) in other fields of 
study. Student experience with online courses prior to COVID-19 was approximately even, with 48.0% 
(2,311) having previously taken an online course and 52.0% (2,507) with no prior online course 
experience. 


Following survey development, several dimensions for each stakeholder group were identified as valuable for 
gathering more in-depth and nuanced information, and interview protocols were developed for each group. 
Interviews were semi-structured, with each protocol containing background information, a number of questions 
organized by dimension, and with opportunities for participants to provide detailed information as well as any 
additional comments. 


Following the close of the spring term (after the initial COVID response) NELC staff selected participants and 
conducted interviews via Zoom, with a few respondents completing their interviews in writing. Interviews (Zoom 
links and/or documents) were uploaded to a shared drive, and videos were auto-transcribed using NVIVO 
transcription. Interviews were then coded using case classifications to identify participant characteristics and 
gather baseline data on interview questions. In-text coding was completed by dimension and subdimension to 
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provide detailed information about participants’ attitudes, opinions, beliefs, and experiences with interview 
questions. 


Interview participants included 13 public university administrators, 10 private university administrators, 20 faculty 
members, and 20 students. 


Quantitative and qualitative results were analyzed to develop recommendations aligned to the Future Action 
Framework and organized by dimension and subdimension, and to identify target audience(s), readiness, needs, 
initiatives, goals, actions, critical success factors, and key performance indicators as appropriate. 


Recommendations 


The Higher Education study resulted in description of both survey and interview 
findings, including a number of recommendations. 


Recommendations are organized by dimension, subdimension, and target audience, and include all relevant 
factors including needs, initiatives, goals, actions, critical success factors, and key performance indicators. 
Recommendations are described below, and are summarized in Appendix IV: Future Action Framework. These 
recommendations include narrative, target audience, needs, initiatives, goals, action lines, critical success factors 
(to be written as “CSFs” in all following tables), and key performance indicators (to be written as “KPIs” in all 
following tables). 


Dimension |: Leadership 


The Leadership dimension measures seven subdimensions including Governance, Strategies, Policies, Processes, 
Resource Allocation, Periodic Review and Updating, and Innovation. 


Within the surveys, this dimension was examined from the perspective of administrators/staff and faculty. 
Students were not asked to rate items from these subdimensions since it would be unlikely that they would have 
insight into these areas. However, there were questions in the student interviews that did touch on the online 
learning strategy and resources. 


Prior to COVID, satisfaction measures in the leadership dimension indicate overall congruence among 
administrators, staff, and faculty, and this largely remained following COVID. The study revealed three primary 
areas that would have benefitted from additional readiness prior to the pandemic: online learning strategy, 
policies, and resource allocation. These items are detailed below, and results indicate that priority was placed on 
these readiness items quickly, as each showed increases in satisfaction post-COVID. 
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Through the interviews that were conducted, more depth of information was acquired regarding the institution’s 
strategy around online learning as well as the sufficiency of resources to adequately support and sustain online 
learning. Within the interviews for the Leadership dimension, it was noted that there is a wide variance in the level 
of preparedness found at the institutions within the Kingdom of Saudi Arabia. A definite difference was noted 
regarding the needs identified by administrators/staff, faculty and students. While not identified directly through 
the interview questions, these differences may be due in part to communications and level of involvement in 
online activities by the various parties. 


Subdimension: Governance 


Measures of satisfaction with governance both pre- and post-COVID, among both administrators/staff and faculty 
remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied (Figure 
1). 


Governance 
50.0% 
40.0% 
30.0% 
20.0% 
10.0% gpg 
Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied 
Admin/Staff/Faculty Satisfaction - Pre-Covid mw Admin/Staff/Faculty Satisfaction - Post-Covid 
Figure 1 


A review of the findings for the Leadership dimension reflect that the majority of survey participants agree or 
strongly agree that their institution has the appropriate governance structure in place to support online 
education. Specifically, two questions reflected strong agreement. This included 79.9% of administrators/staff 
and 80.8% of faculty members indicating that they agreed or strongly agreed with the institution having a 
“governance structure to enable systematic and continuous improvement related to the administration of online 
education” (Figure 2). 
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The institution has a governance structure to enable 
systematic and continuous improvement related to the 
administration of online education. 


50.0% 
40.0% 
30.0% a 
20.0% 
0.0% 
Strongly Disagree Neutral Agree Strongly Agree 
Disagree 


mAdmin/Staff m Faculty 


Figure 2 


In addition, there were 79.5% of administrators/staff and 80% of faculty members that indicated that the 
governance structure enables “clear, effective, and comprehensive decision making related to online education” 
(Figure 3). 


The institution has a governance structure to enable 
clear, effective, and comprehensive decision making 
related to online education. 


50.0% 
40.0% : 
30.0% 
20.0% : 
0.0% 
Strongly Disagree Neutral Agree Strongly Agree 
Disagree 


mAdmin/Staff Faculty 


Figure 3 


However, some variance was noted in terms of faculty involvement in the review process. Administrators/staff 
indicated a stronger agreement (72.7% agreed or strongly agreed) when asked if “the institution has a governance 
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structure in which decisions related to online learning can be reviewed by faculty” (Figure 4). While the faculty 
responses reflected lower agreement (61.2% agreed or strongly agreed) to this question, the overall responses are 
still seen as mostly favorable since only 17.5% indicated that they disagreed or strongly disagreed with the 


statement. 
The institution has a governance structure in which 
decisions related to online learning can be reviewed by 
faculty. 
50.0% 
40.0% 
30.0% 
ZZ 
Strongly Disagree Neutral Agree Strongly Agree 
Disagree 
Admin/Staff Faculty 
Figure 4 


‘Recommendation: Provide sufficient opportunities for faculty to be involved in decisions 
related to online learning. 


Target Audience Administrators at each higher education institution (specifically should be at the 
level that has oversight for faculty) 


Needs Online Learning Governance 

Initiatives Provide opportunities for faculty to be involved in decision making related to online 
learning 

Goals Establish a committee or advisory group for online learning made up of faculty or 


include faculty in an existing committee 


Actions e Form committee structure including mission and objectives 

e Identify appropriate faculty to serve on committee (through election, 
invitation, etc.) 

e Include administrators on committee as appropriate 

e Identify opportunities for faculty to make decisions on curriculum and 
other needs for online learning 

e Define how decisions will be made by the committee 


CSFs Faculty are actively engaged in decision making for online learning 


KPls e Faculty involvement in decision making 
e Time to make decisions 


e Success of decisions made over longer term 
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Subdimension: Strategies 


Measures of satisfaction with strategy both pre- and post-COVID, among both administrators/staff and faculty 
remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied (Figure 
5). 


Strategies 
50.0% 
40.0% 
30.0% 
20.0% 
_, == 
Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied 
m Admin/Staff/Faculty Satisfaction - Pre-Covid mw Admin/Staff/Faculty Satisfaction - Post-Covid 
Figure 5 


Including online learning in the institutional strategy is important in order to ensure that adequate resources are 
allocated. When considering the question on whether “the institution has developed and articulated an 
institutional strategy for online learning while aligning with existing university- and program-level plans (e.g., 
university strategic plans, university and program technology plans, and academic program plans)”, the majority 
of responses (Figure 6) reflected agreement. While more administrators/staff indicated that they agreed or 
strongly agreed (72.9%), the faculty responses were also favorable (67.9% agreed or strongly agreed). 
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The institution has developed and articulated an 
institutional strategy for online learning while aligning 
with existing university- and program-level plans. 


50.0% 
40.0% 
30.0% 
20.0% 
10.0% 

0.0% 


Figure 6 


_~-sesia 


Strongly Disagree Neutral Agree Strongly Agree 
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Admin/Staff mi Faculty 


‘Recommendation: Include online learning in strategic plans at the institutional level in 
order to appropriately align resources. 


Target Audience Institutional leadership 

Needs One or more items related to online learning within the institutional strategic plan 
Initiatives Prioritize online learning 

Goals Prioritize online learning by including it in the strategic plan 

Actions Review existing strategic plan and revise to include online learning where appropriate 
CSFs Commitment to online learning 

KPIs Inclusion of online learning in institutional strategic plan 


Expert Recommendation: Develop a strategic plan for online learning. 


Target Institutional Leadership 
Audience 
Needs Online learning strategic plan 
Initiatives Provide specific and actionable items related to online learning 
Goals Develop a strategic plan for online learning 
Actions e Identify committee made up of representatives from throughout the institution to 
serve on committee to develop 
e Write strategic plan 
e Align resources to strategic plan 
e Evaluate effectiveness of strategic plan 
CSFs Sustain online learning 
KPIs e Development of strategic plan 
e Alignment of resources to strategic plan 
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Subdimension: Policies 


Measures of satisfaction with policies both pre- and post-COVID, among both administrators/staff and faculty 
remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied (Figure 
7). 


Policies 

50.0% 
40.0% 
30.0% — 
20.0% 

0.0% 

Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied 
@ Admin/Staff/Faculty Satisfaction - Pre-Covid mw Admin/Staff/Faculty Satisfaction - Post-Covid 
Figure 7 


The majority of participants (69.3% administrators/staff and 59.7% faculty) agreed or strongly Agreed that “the 
institution has policies that encourage the development of online courses and programs (e.g., a policy for 
intellectual property of course materials).” However, it should be noted that administrator/staff responses 
reflected stronger congruence with this question than faculty (Figure 8). The question for this subdimension 
reflected that a minimal percentage of participants (12.5% administrators/staff and 16.0% faculty) disagreed or 


strongly disagreed. 


The institution has policies that encourage the 
development of online courses and programs. 


50.0% 


40.0% 
30.0% 
20.0% 
0.0% 
Strongly Disagree Neutral Agree Strongly Agree 
Disagree 


mAdmin/Staff mi Faculty 
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Figure 8 


Through the administrator and staff interviews that were conducted, it was noted that it will be critical for the 


Kingdom of Saudi Arabia to provide clear policies on expectations around online learning. This will help 


administrators and staff develop effective strategic plans and align resources appropriately to support online 


learning. As stated by an administrator/staff member that was interviewed “We need to achieve policies. National 


policies and the Ministry of Education policies regarding online learning...now there is no policy that allows us to 


provide online degree or distance education programs.” Another indicated “We were hesitant maybe to do online 


teaching. Also, the Ministry of Education doesn’t really approve of online or distance learning and graduate 


studies. But now, since we were forced to do it, we have.” 


Recommendation: At the country level, the National eLearning Center is encouraged to 
develop national policies on the use of online education in higher education that can 
provide universities with guidelines and expectations for a quality learning experience. 


Target National eLearning Center (NELC) 
Audience 
Needs Policies specific to online learning 
Initiatives Establish standards of quality 
Goals e Develop appropriate policies related to online learning to encourage effective 
implementation 
e Develop standards of quality for online learning curriculum and programs 
Actions e Write policies for online learning at the course and program level (e.g., requirements to 
be able to offer online courses/programs, training requirements for faculty/staff, 
student services that need to be available, technical requirements, etc.) 
e Share policies with institutions of higher education 
e Review implementation of online learning against policies 
CSFs Effective implementation of online learning 
KPIs e Policies written and shared 


e Compliance with policies 


Subdimension: Process 


Measures of satisfaction with process both pre- and post-COVID, among both administrators/staff and faculty 


remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied (Figure 


9). 
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Processes 
50.0% 
40.0% 
30.0% 
20.0% 
10.0% 7 fr 
Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied 
m Admin/Staff/Faculty Satisfaction - Pre-Covid w Admin/Staff/Faculty Satisfaction - Post-Covid 
Figure 9 


Processes are an important component to ensuring the success of an online program since they allow for 
standardization. Having a process that includes centralized support is one way to provide for the needs of faculty 
and students while creating a more equitable learning experience. There was a strong congruence among 
administrators/staff (75.9% agreed or strongly agreed) and faculty (76.4% agreed or strongly agreed) in the belief 
that “the institutional structure provides centralized support for online program development, faculty within 
online programs, and students enrolled in online programs” (Figure 10). 


The institutional structure provides centralized support 
for online program development, faculty within online 
programs, and students enrolled in online programs. 


50.0% ‘ 
40.0% 

30.0% 

20.0% - 
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Strongly Disagree Neutral Agree Strongly Agree 
Disagree 


mAdmin/Staff mm Faculty 


Figure 10 
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Expert Recommendation: Ensure that processes (e.g., admissions, registrar requests, 
testing, etc.) work with students at a distance. 


Target Audience 


Administration, faculty 


Needs 


Processes to support virtual learners 


Initiatives 


Processes that meet the needs of online learning 


Goals Implement appropriate processes to support online learners 
Actions e Review existing processes and modify for online learners 
e Identify and develop additional processes needed to support online learners 
CSFs Effective support for online learners 
KPIs Processes are written and implemented 


Subdimension: Resource Allocation 


Measures of satisfaction with resource allocation both pre- and post-COVID, among both administrators/staff and 
faculty remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied. 
It should be noted that a higher percentage of respondents indicated a neutral opinion in this subdimension, and 

neutral responses increased slightly post-COVID (Figure 11). 


50.0% 


40.0% 


0.0% 


Resource Allocation 


30.0% 
20.0% 


Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied 


Admin/Staff/Faculty Satisfaction - Pre-Covid mw Admin/Staff/Faculty Satisfaction - Post-Covid 


Figure 11 


Within the Leadership dimension, the strongest agreement (77.5% of administrators/staff and 82.3% of faculty 


agreed or strongly agreed) could be found related to the subdimension of Resource Allocation with a question on 


“training and development or faculty development programming to provide faculty with the knowledge, skills, 


and abilities to design and teach online courses” (Figure 12). 
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The institution has established training and development 
or faculty development programming to provide faculty 
with the knowledge, skills, and abilities to design and 
teach online courses. 


60.0% 


40.0% 

0 = 
Strongly Disagree Neutral Agree Strongly Agree 
Disagree 


mi Admin/Staff mi Faculty 


Figure 12 


An area of concern was identified with the question “incentives and/or financial compensation are provided to 
faculty for professional development and course design activities” where 49.0% of administrators/staff and only 
26.6% of faculty indicated agreed or strongly agreed, reflecting a potential need to establish a process that 
provided some form of incentives for these activities (Figure 13). 
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There also seemed to be some concern about the instructional designers having “positional parity to faculty 
through faculty status or degrees” with 65.2% of administrators/staff and 53.1% of faculty agreeing or strongly 
agreeing that this was adequate at their institution (Figure 14). 
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The institution has organized the instructional designers 
with positional parity to faculty through faculty status or 
degrees required. 
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Figure 14 


The survey results also indicated that there may be some work to do around the “process of planning and 
allocating resources for online learning, including financial resources, in accordance with strategic planning.” 
There were 66.3% of administrators/staff and 53.1% of faculty that agreed or strongly agreed that their institution 
was prepared with resources (Figure 15). 


The institution has a process for planning and allocating 
resources for online learning, including financial 
resources, in accordance with strategic planning. 
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Figure 15 


Some of those interviewed felt that they had adequate resources, including a faculty member who said “...during 
COVID-19, we didn’t have any problems with the transition...we had good financial support from the 
administration and the IT committee was good enough to handle all of the problems that we face.” Another 
faculty member indicated that in terms of financial, technology, and personnel resources to support online that 
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their institutions had “an elearning center and they’ve been providing almost everything. Support. Training. 
Everything.” However, in many other interviews conducted with higher education administrators/staff, faculty, 
and students, a consistent theme was that there will be a need for personnel, technology and financial resources. 
Although one administrator/staff person interviewed indicated a need for resources and information related “to 
the methodology and pedagogy and also the technology” as well as” the strategy for implementing” online 
learning. 


Administrators/staff, faculty, and students all seemed to agree that there are concerns around consistent and 
reliable access to technology for all students. This concern needs to be addressed on a broader scale. As one 
student said, “I’m not sure about the financial resources, but | think if Saudi Arabia decides to make this a goal 
that they will want to achieve, of course they are going to budget the whole thing for other universities to have the 
financials for it. But as for technical support, | don’t think they’re ready because even in classes, | don’t remember 
having lots of tech support. For the tiniest things, for example, the projectors inside the classrooms, they didn’t 
have much tech support.” The issue with technology was echoed in an interview with a faculty member who 
indicated that “...our real problem is the students that most, or some of them, they have weak computers, weak 
connection. So that’s the challenging part.” 


Recommendation: Provide incentives to faculty to participate in professional 
development and create online courses. 


Target Administrators 

Audience 

Needs Rewards, recognition or other incentives 

Initiatives Encourage faculty to develop online courses 

Goals Provide incentives to faculty to participate in professional development in order to develop 


online courses 


Actions e Determine what types of incentives will be used institutionally (e.g., financial, 
additional resources, recognition, etc.) 

e Create guidelines to receive incentive (e.g., attend certain number of workshops, 
develop first online course, teach online course, etc.) 


CSFs Faculty commitment and participation in professional development 
KPIs e Increase in number of faculty qualified to teach online 


e Increase in number of online courses offered 
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Recommendation: Evaluate internet availability throughout the Kingdom of Saudi Arabia 
to determine opportunities to improve access. Though this can be a bit more challenging 
and is a problem faced by countries around the world, it may be necessary. 


Target Government (possibly regional) 
Audience 
Needs Adequate internet access 
Initiatives Increase accessibility to online learning 
Goals Provide adequate internet access throughout the Kingdom of Saudi Arabia (rural, suburban, 
and urban) 
Actions e Determine weak points for internet access throughout the country 
e Identify locations to install or upgrade internet 
CSFs Learners are able to access online courses 
KPIs e Increase in number of students able to enroll in online learning 


Recommendation: Prioritize access to technology, technology support, and technology 
training to better support students. 


Target Administrators 
Audience 
Needs Technology, Technology Support and Technology Training 
Initiatives Improve use of technology 
Goals Allocate appropriate resources to support the use of technology for online learning 
Actions e Review technology being used to support online learning for effectiveness 
e Provide technology support for students and faculty that encounter issues with being 
able to access their online course 
e Provide professional development or workshops to students and faculty to help more 
effectively use educational technology 
CSFs Learners and faculty are able to more effectively engage with online classroom 
KPIs e Increased use of technology to support online learning 


e Increased access to support for technology issues 
e Increased training for use of required technology 
e Increased student and faculty satisfaction with online learning 


Subdimension: Periodic Review and Updating 


Measures of satisfaction with periodic review and updating both pre- and post-COVID, among both 


administrators/staff and faculty remained stable, with the majority of participants indicating that they are overall 


satisfied or very satisfied. It is notable that the number of respondents indicating that they are very satisfied with 


this subdimension increased post-COVID (Figure 16). 
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In regards to continuous improvement, faculty were asked if “the institution has a governance structure to enable 


systematic and continuous improvement related to the administration of online education.” Having plans in place 


that allow for regular updates to curriculum is always important in higher education, but moreso in online 


learning where information provided to students needs to be current, relevant and keep students engaged. In 


response to this question, the majority of faculty (66.8%) indicated that they agreed or strongly agreed. There 


were no questions for administrators/staff included for this subdimension. 


Expert Recommendation: Conduct regular institutional reviews of online learning courses 
and programs. 


Target Administrators, Faculty 
Audience 
Needs Policies, processes and procedures specific to the regular review of online learning courses 
and programs 
Initiatives Development of policies, processes and procedures to review and provide feedback on online 
learning courses and programs 
Goals Maintain and regularly update the quality of online courses and programs 
Actions e Develop and implement policies, processes and procedures specific to the regular 
review of online learning courses and programs 
e Use standardized rubrics and scorecards (Online Learning Consortium, Quality 
Matters) as part of the process to review against quality standards 
e Conduct regular reviews as part of continuous improvement process 
CSFs Online learning courses and programs continue to meet best practices 
KPIs e Online learning courses and programs reviewed per policy 


e Online learning courses and programs continuously updated 
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Expert Recommendation: Conduct regular and substantive reviews of online learning at 
the National level. 


Target National eLearning Center (NELC) 
Audience 
Needs Policies, processes and procedures specific to the regular review of institutional online 


learning programs 


Initiatives 


Development of policies, processes and procedures to review and provide feedback to 
institutions on online learning courses and programs 


Goals Ensure the quality of online learning programs offered at institutions 
Actions e Develop and implement policies, processes and procedures specific to the regular 
review of online learning programs 
e Provide feedback on quality of program compared to industry standards 
CSFs Quality online learning programs are offered within the Kingdom of Saudi Arabia 
KPIs e Policies, processes and procedures developed and shared with institutions 


e Regular reviews of online learning programs conducted 


Subdimension: Innovation 


Measures of satisfaction with innovation in the leadership dimension both pre- and post-COVID, among both 
administrators/staff and faculty remained stable, with the majority of participants indicating that they are overall 
satisfied or very satisfied. It is notable that the number of respondents indicating that they are very satisfied with 
this subdimension increased post-COVID (Figure 17). 
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Under the Leadership dimension, the Innovation subdimension appears to have the most opportunity for 


improvement. There were two questions on the survey for administrators/staff and faculty that focused on this 
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area. The first sought agreement with whether “the institution provides financial incentives and support resources 
to support resources to advance pedagogical and technological innovation.” Administrators/staff indicated 
stronger agreement (51.0% agreed or strongly agreed) with this statement than faculty (33.6% agreed or strongly 
agreed). Figure 18 shows the results for this question and reflects 30.4% of faculty indicating that this is an area of 
concern. 
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Figure 18 


The second question on the survey related to the subdimension of Innovation did show slightly stronger 
agreement with 48.3% of administrators/staff and faculty indicating that they agreed or strongly agreed when 
asked if “institutions include the aspect of innovation in teaching and learning as part of an award system and/or 
as part of the evaluation and promotion policies (e.g., award for innovation in teaching and learning; promotion 
review includes measure of innovation in teaching).” Once again, administrators/staff showed higher congruence 
with a total of 57.1% that agreed or strongly agreed while faculty responses showed lower agreement with 45.0% 
agreed or strongly agreed (Figure 19). 
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Figure 19 


Recommendation: Rewards, recognition or other incentives should be provided to 
encourage innovation. 


Target Administrators 

Audience 

Needs Rewards, recognition or other incentives 

Initiatives Encourage faculty to be more pedagogically or technologically innovative 

Goals Provide incentives to faculty to participate in professional development in order to develop 


online courses 


Actions e Determine what types of incentives will be used institutionally (e.g., financial, 
additional resources, recognition, etc.) 

e Create guidelines to receive incentives (e.g., attend certain number of workshops, 
develop first online course, teach online course, etc.) 


CSFs Faculty commitment to innovation and participation in professional development 


KPIs e Increase in number of faculty engaging in pedagogical innovation 


e Increase in number of faculty engaging in technological innovation 
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Dimension II: Curriculum Design & 
Planning 


The Curriculum Design and Planning dimension measures five subdimensions including Instructional Design 
Methods & Universal Design for Learning (UDL), Alignment with Standards, Course Syllabi, Course Materials and 


Content, and Innovation. 


Prior to COVID, satisfaction measures in the curriculum design and planning dimension indicate overall 
congruence among administrators, staff, and faculty, and this largely remained, though there were decreases in 
satisfaction and increases in neutral responses following COVID. As supported by survey and interview data, 
readiness needs were primarily focused on preparedness for fully online course delivery, including design, syllabi, 
and materials and content. Results suggest that pre-COVID delivery of these items was satisfactory, but not 
sufficient to support fully online courses. 


This dimension primarily queried faculty and students about their actions, experiences, and perceptions of course 
design and planning. Administrators/staff were asked about responsibility, resources, and curriculum design and 
planning components, and responses were favorable overall, consistently indicating above 70% that effective 
practices are followed. Qualitative data further indicate that while there are some areas in which administrators 
would like to see improvement (detailed in subdimensions), they are largely positive about this dimension. 


As might be expected due to their deep engagement at a course level, there were some differences in faculty 
responses to mutual questions, with faculty regularly indicating at or above the administrator/staff agreement 
level that essential curriculum design and planning components are included in courses. One primary difference 
between administrators/staff and faculty emerged. When asked about their agreement with the statement that 
“faculty should be included in the development and decision-making process for online curriculum,” 
administrators/staff indicated at 73.1% that they agreed or strongly agreed, and faculty agreed or strongly agreed 
at 78.3%. There was a slight difference in the rate of disagree/strongly disagree responses, 10.0% and 8.4% 
respectively. These differences are not large, but when coupled with interview data, bear some consideration as 
discussed below. 


Students indicated a high level of agreement with administrators/staff and faculty overall regarding curriculum 
design and planning components, with two findings of particular interest. There was a discrepancy between these 
groups with regard to consistency in course structure, with students indicating at a rate 63.4% (ten percentage 
points below administrators/staff and faculty) that this has been their experience, and 20.0% disagreeing or 
strongly disagreeing. Other noticeable decreases in student perceptions or experiences as compared to 
administrators/staff and/or faculty are found in survey items assessing active learning and interaction as well as 
the facilitation of learning materials and activities. These items bear consideration and are discussed in 
recommendations below. 
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Subdimension: Instructional Design Methods & 
Universal Design for Learning (UDL) 


Measures of satisfaction with instructional design methods and Universal Design for Learning both pre- and post- 
COVID, among both administrators/staff and faculty remained stable, with the majority of participants indicating 
that they are overall satisfied or very satisfied (Figure 20). 


Instructional Design Methods & 
Universal Design for Learning 
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Findings indicate that administrators/staff and faculty agree that there are “processes in place to ensure effective 
development and implementation of curriculum and design.” As illustrated by both a public and private university 
administrator, the focus of improvements to instructional design methods and UDL is on both curriculum 
development and quality. One stated, “We would have some effort, some time, some knowledge in order to help 
us move into or incorporate different types of degrees... Also, objectives might be also adjusted, but we would 
have to modify the current models into online teaching models.” and the other noted “As | said that we had 
multiple strategic items. We only focused and we have achieved the first one. But the second one is actually 
related to this issue where we need the people to have more quality. | guess there is fluctuation on the results 
between the universities...but we are using the international standards that are known by experts as an indication 
of technology, quality models, all of them.” 


As noted above, though the percentage discrepancy is relatively small, a difference did emerge in the survey data 
indicating that faculty believe at a higher rate than administrators (78.4% versus 73.1%) that they should be 
included in curriculum design and development. It is recommended that opportunities for faculty input be 
provided regularly, and that clear communication regarding decision making for these areas be provided 
frequently to ensure buy-in across the institution(s) and effective implementation of decisions. 
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Notably, in the qualitative data, a clear theme emerged in which faculty from both public and private institutions 
expressed a need for training and assistance with instructional design. Though administrators expressed in 
interviews that these opportunities are present, it is recommended that additional opportunities be developed, 
offered at regular intervals, and that ongoing support for instructional design be present. 


In the subdimension of instructional design and UDL, students indicated that their experiences have not been 
consistent (63.4% agreed or strongly agreed and 20% disagreed or strongly disagreed), while administrators/staff 
and faculty believe at or above 73.3% that design and organization is coherent across courses (Figure 21). Some 
inconsistency is likely due to remote teaching as a result of COVID-19, but it is recommended that instructional 
design be included in course reviews for quality and that whenever possible, consistent practices be employed to 
ensure a consistent and coherent experience for optimal student learning. 


The instructional design includes a consistent course 
structure, manageable units, and organization across 
courses for coherency of student experience. 
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Recommendation: Provide regular opportunities for faculty input on curriculum design 
and planning decisions, and clearly communicate with faculty regarding such decisions to 
ensure buy-in across the institution(s) and effective implementation of decisions. 


Target Faculty 
Audience 
Needs e Opportunities for input on curriculum design and planning decisions 


e Communication regarding such decisions 


Initiatives e Create opportunities for faculty participation in decision-making, such as committee 
involvement and soliciting written input and/or feedback 

e Regularly communicate with faculty regarding curriculum design and planning 
decisions 


Goals To utilize faculty expertise by including them in curriculum design and planning decisions and 
to increase the awareness and understanding of such decisions 


Actions Create opportunities and communications plans for faculty involvement in curriculum design 
and planning decisions 


CSFs e Presence and awareness of opportunities for faculty participation 
e Regular communications to faculty regarding decisions 


KPIs e Faculty are aware of and participate in curriculum design and planning decision- 
making opportunities 
e Communications are created and sent regularly, and faculty indicate awareness and 


understanding of decisions 


Recommendation: Develop and regularly offer professional development and training on 
instructional design and UDL, and provide ongoing instructional design support to 
faculty. 


Target Faculty 
Audience 
Needs e Professional development and training on instructional design and UDL 


e Ongoing instructional design support for faculty 


Initiatives e Develop and regularly offer professional development and training focused on 
instructional design and UDL 
e Develop ongoing instructional design support resources for faculty 


Goals Increase faculty knowledge, skill, and support resources on instructional design and UDL 


Actions Develop and regularly offer professional development and training on instructional design and 
UDL, and ensure that faculty are aware and able to access both training and support options 


CSFs e Presence of professional development and support opportunities 


e Awareness of professional development and support opportunities 
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KPIs 


Faculty participation in professional development opportunities 
Faculty use of support resources 
Evaluation of effectiveness based on course design outcomes and faculty feedback 


Recommendation: Include instructional design in quality review processes and whenever 
possible, employ standardized practices. 


Target Faculty, Students 
Audience 
Needs Standardized course design practices 
Initiatives e Develop and implement standardized instructional design practices 
e Include instructional design in quality reviews 
Goals Increase course consistency and coherence through the use of standardized practices and 
quality review 
Actions e Develop standardized instructional design practices 
e Ensure that faculty understand and implement these practices 
e Include these practices in quality course reviews 
CSFs e Standardized instructional design practices are developed 
e Standardized instructional design practices are implemented 
e Quality course reviews include instructional design 
KPIs e Instructional design standards are evident in courses 


Quality course reviews reflect adherence to standardized instructional design 
Student feedback indicates an increase in consistent and coherent design across 
courses 


Subdimension: Alignment with Standards 


Measures of satisfaction with alignment with standards both pre- and post-COVID, among both 


administrators/staff and faculty remained stable, with the majority of participants indicating that they are overall 
satisfied or very satisfied (Figure 22). 
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Both survey and interview data indicate that administrators and faculty/staff believe that courses are aligned with 
best practices and quality standards for curriculum design and planning. They indicate at high levels (74.3% and 
76.7%, respectively) that standards for design, development, and delivery are employed. This subdimension 
measured student perceptions indirectly (as they may not be explicitly aware of alignment standards), and their 
response, 65.1% agreeing or strongly agreeing and 18.6% disagreeing or strongly disagreeing, that they were “able 
to develop the necessary knowledge and skills to achieve learning outcomes” (Figure 23) may improve with 
further review and alignment of courses according to standards. 
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Recommendation: Ensure alignment with standards through quality review, and collect 
student feedback and measure their learning outcomes to improve and monitor 


effectiveness. 
Target Faculty, Students 
Audience 
Needs Ensure that courses are aligned with standards and that students are able to develop the 
knowledge and skills needed to achieve learning outcomes 
Initiatives e Quality course review for alignment with standards 
e Collect student feedback on their ability to achieve learning outcomes 
e Regularly monitor student outcomes in courses 
Goals Increase alignment to standards in courses to support successful student learning outcomes 
Actions e Include alignment with standards in quality course reviews 
e Provide opportunities for student feedback regarding their ability to achieve learning 
outcomes 
e Measure student outcomes in courses for continuous improvement 
CSFs e Alignment with standards is reflected in quality course reviews 
e Students are able to provide feedback regarding their ability to achieve learning 
outcomes 
e Student outcomes in courses are measured and reviewed regularly 
KPIs e Alignment with standards is evident in quality course reviews 


Students participate in feedback opportunities and indicate areas of success or 
challenge with their ability to achieve learning outcomes 

Measurement and review of student outcomes in courses reflects improvement 
and/or is used for continuous improvement 


Subdimension: Course Syllabi 


Measures of satisfaction with course syllabi both pre- and post-COVID, among both administrators/staff and 


faculty remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied 


(Figure 24). 
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Figure 24 


All three groups, administrators/staff, faculty, and students overall agreed that syllabi contain essential course 
and content information (Figure 25). Faculty displayed a particularly high rate of agreement (84.2%) and while 
students did agree, their responses indicate the potential for increased syllabi effectiveness that could be 
addressed through course design. Ensuring that syllabi are easily located and contain clearly labeled course 
objectives, learning outcomes, assessment methods, and descriptions of content and activities could improve the 
student learning experience. Of most note in this subdimension, students indicated that expectations for 
participation are not included in syllabi at as high a rate of other components, and the explicit inclusion of these 
expectations could benefit students and improve their learning experiences. 
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Recommendation: Develop and implement clear syllabi components, including clearly 
labeled course objectives, learning outcomes, assessment methods, descriptions of 
content and activities, and expectations. Ensure that syllabi are easy to locate and use in 
a consistent course design practice. 


Target Faculty, Students 
Audience 
Needs e Syllabi contain standard components that are clearly labeled and include course 


objectives, learning outcomes, assessment methods, descriptions of content and 
activities, and expectations. 
e Syllabi are easy to locate and use with consistency in course design 


Initiatives e Develop and implement standard syllabi components 
e Implement course design standards ensuring that syllabi are consistently easy to 
locate and use 


Goals Increase students’ ability to consistently locate and use syllabi, and increase students’ 
understanding of course components and expectations through standard syllabi development 
practices 

Actions e Develop and implement standard syllabi components 


e Develop and implement standard course design practices for the location of syllabi 
within the course 

CSFs e Standard syllabi components are developed and implemented 

e Standard course design practices for the location of syllabi within the course are 


developed and implemented 
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KPIs e Quality course reviews indicate that standard syllabi components are used and that 
syllabi are placed appropriately within the course design 

e Student feedback indicates that syllabi components are present and identifiable, and 
that syllabi are consistently located within their courses 


Subdimension: Course Materials and Content 


Measures of satisfaction with course materials and content both pre- and post-COVID, among both 
administrators/staff and faculty remained stable, with the majority of participants indicating that they are overall 
satisfied or very satisfied. It is notable that post-COVID, there was a slight decrease in those reporting that they are 
very satisfied, and an increase in neutral responses for this subdimension (Figure 26). 
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Responses in the Course Materials and Content subdimension indicate that administrators/staff, faculty, and 
students consistently agreed that materials are “accessible, easy to use, and may be accessed on a variety of 
devices.” This is a key component of effective dissemination and use of materials. Other items rated favorably 
include the use of a variety of content/materials types that are appropriate for multiple learning objectives, 
including critical reflection, analysis, and real-world applications. Two areas in this subdimension are 
recommended for review and potential improvements as rated by students: ensuring sufficient breadth, depth, 
and currency of instructional materials, and including current content such as online resources and/or videos. For 
each of these items, approximately 65% of students agreed that these elements are present, with approximately 
18.5% disagreeing. It is recommended that course materials and content be regularly reviewed for sufficient 
coverage and currency, and be updated as appropriate with current and media-rich content. 


Recommendation: Regularly review course materials and content for sufficient coverage 
and currency, and update as appropriate with current and media-rich content. 
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Target Faculty 

Audience 

Needs Ensure that course materials and content include sufficient coverage and are regularly 
updated with current and media-rich content 

Initiatives e Regularly review course materials and content 

e Regularly update course materials and content with current and media-rich sources 

Goals Continually ensure that content is sufficient, updated, and includes current and media-rich 
resources 

Actions Develop regular review and updating processes and schedules for course content and 
materials 

CSFs Course content and materials are regularly reviewed for currency and resource content and 
updated as appropriate 

KPIs Reviews of course content and materials are completed and resources are updated with 


current and media-rich content 


Expert Recommendation: Ensure academic rigor of content with activities at the 
appropriate level of Bloom’s Taxonomy. 


Target Faculty 
Audience 
Needs Ensure that course content (activities, assignments) are at an appropriate level of academic 
rigor and include higher levels of thinking 
Initiatives Provide academic rigor in all courses 
Goals Offer academically rigorous programs that appropriate prepare learners 
Actions e Review activities and assignments in courses to ensure they provide for higher levels 
of thinking (Bloom’s Taxonomy) 
e Add activities and assignments that provide for higher order thinking 
CSFs Course content meets appropriate level of academic rigor 
KPIs Activities and assignment included in courses offer higher order thinking 


Subdimension: Innovation 


Measures of satisfaction with innovation in curriculum design and planning both pre- and post-COVID, among 
both administrators/staff and faculty remained stable, with the majority of participants indicating that they are 
overall satisfied or very satisfied. Neutral responses for this subdimension appear to be more frequent than in 
other areas (Figure 27). 
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Survey and interview data indicate that administrators/staff and faculty agree that their institutions support 
innovation in curriculum design and planning. In one example of continued innovation, a faculty interview 
participant indicated that their institution goes beyond the course level: “We are doing our study plan and at the 
same time, we are having personal development, where we provide more extra activities, where we train them on 
leadership, on time management... and that’s part since most are living here with us and they live in the dorm. We 
provide them with so many activities, actually.” 


Students indicated that their perceptions of innovation, particularly regarding teaching methods and technology, 
were lower, with 60.2% agreeing or strongly agreeing, and 23.7% disagreeing or strongly disagreeing. It is 
recommended that in addition to the existing innovative planning and practices, student feedback relevant to 
innovation be included in curriculum design and planning. 


Recommendation: Provide opportunities for student feedback regarding innovative 
pedagogical and technological practices in curriculum design and planning. 


Target Students 

Audience 

Needs Include student feedback in innovative pedagogical and technological practices 

Initiatives Provide opportunities for students to give feedback on implemented practices, and for them to 
share their thoughts, needs, and ideas on innovation 

Goals Increase student awareness of innovative practices and incorporate their feedback as 
appropriate in continual improvement of curriculum design and planning innovation 

Actions Develop and implement opportunities for student feedback on the success and development 
of innovative practices in curriculum design and planning 

CSFs Student feedback opportunities are provided 

KPIs Student feedback is evaluated and incorporated into decision-making on innovative practices 
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Dimension III: Online Teaching & Learning 


The Online Teaching & Learning dimension measures seven subdimensions including communication, 
engagement, expectation setting, outcomes, course interaction, feedback, teaching and learning resources, and 
innovation. 


Prior to COVID, satisfaction measures in the online teaching and learning dimension indicated positive responses, 
through consistent decreases in satisfaction and increases in neutral responses post-COVID as well as survey and 
interview data indicate that readiness for fully online teaching and learning was not met across all subdimensions. 
However, it is clear that immediate efforts were made to address key areas. 


Overall findings indicate that while the data was generally positive about perceptions and experiences with online 
teaching & learning related to the pandemic, administrators/staff, faculty, and students all had slightly different 
interpretations of their value and success. 


The data in this dimension, both quantitative and qualitative, indicates a successful emerging online support 
experience. Students, faculty, and staff all reported largely positive experiences and perceptions, and the goals 
and outcomes of faculty and students seemed to be aligned in regard to expectations related to the learning 
experience. To that end, recommendations for the online teaching & learning dimension revolve around shared 
understanding and process infrastructure so that the successes already observed can be identified, articulated, 
and operationalized. 


Subdimension: Communication 


Measures of satisfaction with communication and planning both pre- and post-COVID, among both 
administrators/staff and faculty indicate that the majority of participants responded that they are overall satisfied 
or very satisfied. However, the number of those indicating very satisfied decreased markedly post-COVID, and 
neutral, unsatisfied, and very unsatisfied responses increased (Figure 28). 
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Figure 28 


Students felt that faculty did a good job of communicating to them in the virtual setting, with 67.9% agreeing or 
strongly agreeing that faculty communicated with them in ongoing, timely, effective and appropriate ways. This 
seems directly tied to expectations as reported by administrators, staff, and faculty. Eighty-one percent of 
administrators and staff and 89.6% of faculty report that they are expected to hold virtual office hours and 
frequently communicate with their students via e-mail, course news and announcements, online discussions, and 
assignment feedback. 


When it comes to clarity of communication, the data indicates there may be some confusion among faculty, staff, 
and students in regard to terminology as well, which could cause confusion in the expectations individuals have 
for what good communication means. “Distance learning, you hear that? Do you want learning? You hear different 
things. What’s the meaning of this term? Well, some of us will say, OK, putting a lecture online that’s kind of 
distance learning. Having the e-mail and communicating with my students...we need to define the terminology of 
distance.” 
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Recommendation: Incorporate shared definitions into faculty development and student 
support. As a part of a larger online learning strategy, consider identifying and defining a 
list of core definitions as they apply to the institution. Consider what measurements (if 
any) define these terms. For example, does “online learning” mean fully asynchronous, or 
merely delivered fully online? What percentage of seat time is replaced in order fora 
course to be defined as “hybrid?” 


Target Administrators/Staff, Faculty 

Audience 

Needs Identify and define a list of core definitions as they apply to the institution 

Initiatives Consider the components and measurements associated with terms used for various iterations 
of online learning, identify, and define terms that will be consistently used 

Goals Increase the clarity and effectiveness of communication regarding the various iterations and 
components associated with online learning 

Actions e Determine terminology needs for each iteration or component associated with online 

learning 


e Define metrics (as appropriate and/or needed) and definitions of use for each term 
e Communicate terms and definitions, and implement consistent use of selected 
terminology across all stakeholder groups 


CSFs e Terminology is selected and defined 

e Metrics associated with terminology (as appropriate) are selected and defined 
e Selected terms and definitions are communicated to stakeholders 

e Consistent use of selected terms is implemented 


KPIs Selected terminology is consistently used across stakeholder groups and accurately used for 
various online learning instances 


Subdimension: Engagement 


Measures of satisfaction with engagement both pre- and post-COVID indicate that the majority of participants 
responded that they are overall satisfied or very satisfied. It is of note that post-COVID, the number of respondents 
indicating satisfied or very satisfied decreased, while neutral, unsatisfied, and very unsatisfied responses 
increased (Figure 29). 
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This subdimension tells a mixed story. Administrators/staff and faculty largely agreed that course design 
promoted student engagement “including academic challenge and social involvement with each other and the 
instructor.” Meanwhile, it seems the student experience aligns with this perception, and that faculty and online 
teaching & learning support staff are paying attention. A high volume of students, 81.3% agreed or strongly agreed 
that student interactions with other students and with the instructor were facilitated in a variety of ways. 


In this dimension, students are a little more neutral with only 59.7% agreeing or strongly agreeing with the 
statement “students were academically challenged and interacted with each other and the faculty.” 
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Recommendation: Provide interactive opportunities, both formal and informal, for 
students to interact with one another and faculty, and ensure that academic challenge is 
incorporated into these experiences as appropriate. 


Target Faculty, Students 

Audience 

Needs Increase opportunities for students to interact with their peers and faculty, including 
academically challenging components 

Initiatives Implement both formal and informal opportunities for students to interact with each other and 
faculty, and to engage with academically challenging interactivity 

Goals Increase student engagement with course content, their peers, and faculty, including 
academically challenging components 

Actions Include intentional opportunities for academically challenging interactions with students and 
faculty in course design and delivery 

CSFs Online courses provide interactive opportunities for students and the peers as well as faculty, 
and these opportunities include academically challenging components 

KPIs e Quality course reviews indicate the presence of academically challenging interactive 


opportunities 
e Student evaluations and feedback indicate that interactive opportunities are present 
and academically challenging 


Subdimension: Expectation Setting 


Measures of satisfaction with expectation setting both pre- and post-COVID, among both administrators/staff and 
faculty reflect that the majority of participants are overall satisfied or very satisfied. It is of note that both satisfied 


and very satisfied responses decreased post-COVID while neutral, unsatisfied, and very unsatisfied responses 


increased (Figure 30). 
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Expectations are important in online courses, and 67.2% students agreed or strongly agreed that their courses 
contained clearly defined student activity expectations. Faculty (80.3%) overwhelmingly believe that barriers can 


be reduced in courses by providing clear expectations. Interview data supported the survey data in this 
subdimension. “I think expectations need to be clear,” said one respondent. “I think expectations with your 
student. You need to be present for these courses.” Data further indicates that faculty are designing courses in 
ways that expectations are clear. Students agreed or strongly agreed at a rate of 72.2% that the expectations for 


student assignment completion were provided. 


Recommendation: Ensure that online courses include clearly defined student activity 
expectations, and that these expectations are communicated to students. 


Target Faculty 
Audience 
Needs Expectations for student activity are clearly defined and communicated to students 
Initiatives Develop, define, and communicate expectations for student activity 
Goals Increase presence, communication, and awareness of student activity expectations to 
support successful student learning 
Actions e Develop and define student activity expectations 
e Communicate expectations to students 
CSFs e Student activity expectations are present 
e Student activity expectations are communicated to students 
KPIs e Student activity expectations are evident in quality course review 


e Student evaluations and feedback indicate that expectations are present an 
effectively communicated 


Expert Recommendation: Provide information on expectations for teacher 


responsiveness. 

Target Faculty 

Audience 

Needs Expectations for responses from the faculty are clearly defined and shared with students 

Initiatives Establish expectations for responsiveness 

Goals To set expectations regarding when students can expect responses from the teacher 

Actions Develop, define and communicate expectations for when students can expect responses (to 
emails, phone calls, etc.) from the teacher 

CSFs Established parameters for when teachers will respond to student communications 

KPIs Teachers responding to student communications within set parameters 
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Subdimension: Outcomes 


Measures of satisfaction with outcomes both pre- and post-COVID, among both administrators/staff and faculty 
reflect that the majority of participants are overall satisfied or very satisfied. It is of note that both satisfied and 
very satisfied responses decreased post-COVID while neutral, unsatisfied, and very unsatisfied responses 
increased (Figure 31). 


Outcomes 
50.0% 
40.0% 
30.0% 
20.0% 
0.0% ’ 
Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied 
Admin/Staff/Faculty Satisfaction - Pre-Covid mw Admin/Staff/Faculty Satisfaction - Post-Covid 
Faculty 31 


While most survey and interview respondents do see the outcomes of online learning as comparable to face-to- 
face outcomes, others disagree. These results are distinctly mixed, and remain the focus of intense debate in 
digital education spheres. For example, while 59.3% of students responded in the survey that they agreed or 
strongly agreed that online courses had comparable outcomes to onsite courses, 27% disagreed or strongly 
disagreed with the same. Only 13.8% of students claimed they were neutral. 


Administrators and staff had more positive perceptions than the students experiencing them, with 65.9% agreeing 


or strongly agreeing that online courses have comparable outcomes to onsite courses, with 17.3% disagreeing or 


strongly disagreeing (Figure 32). Faculty were the least likely to view online course experiences as comparable to 


onsite ones, with 57.5% saying they agreed or strongly agreed, and 25.2% saying they disagreed or strongly 
disagreed. 
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Online courses had comparable outcomes to onsite 


courses. 
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Among those that disagreed, perceived gaps in technology for teaching and assessing hands-on medical and 
nursing concepts seemed to be the primary driver of the opinion. “We are trying to like compensate and cover 
some learning outcomes through virtual teaching,” said one respondent. “But you’re not always able to achieve 
all the outcomes through the virtual teaching. So this is the main two areas, | think we would have some 
challenges, assessment and how we teach skills through virtual.” 
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Recommendation: Measure, review, and communicate student learning outcomes for 
online courses, and if appropriate and/or useful, include communicating comparisons to 
student learning outcomes for onsite courses. 


Target Administrators/Staff, Faculty 
Audience 
Needs Measure, review, and communicate student learning outcomes in online courses, and provide 


comparative onsite data as appropriate 


Initiatives e Develop and implement student learning outcome metrics for online courses 
e Communicate findings, including onsite comparisons as appropriate 


Goals Develop and communicate an understanding of student learning outcomes in online courses, 
including comparative onsite data as appropriate, to aid in both continual course improvements 
as well as to provide accurate perceptions of the efficacy of online courses 


Actions e Develop student learning outcome metrics for online courses 

e Measure student learning outcomes in online courses 

e Communicate findings, including comparative onsite data as appropriate 
e Use findings to aid in continual improvement of courses 


CSFs e Student learning outcome metrics are developed and implemented for online courses 
e Results of these metrics and comparative data are communicated 
e Findings are utilized in continual improvement processes 


KPIs e Student learning outcomes for online courses are measured 

e Results are communicated and include relevant comparisons to onsite courses 

e Findings result in continual improvement for online courses 

e Administrators/staff and faculty indicate that they understand student learning 
outcomes data for online courses and their perceptions regarding the efficacy of online 


courses are in alignment with findings 


Expert Recommendation: Provide rubrics for all assignments. 


Target Faculty 

Audience 

Needs Rubrics for all students learning activities and assignments 

Initiatives Clearly articulate criteria for assignments 

Goals To provide students with clearly defined criteria for the grading of all assignments 
Actions Develop rubrics linked to student learning outcomes to show grading criteria 
CSFs Increased awareness of grading criteria 

KPIs Rubrics developed and provided for all assignments and activities 
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Subdimension: Course Interaction 


Measures of satisfaction with course interaction both pre- and post-COVID, among both administrators/staff and 
faculty reflect that the majority of participants are overall satisfied or very satisfied. It is of note that both satisfied 
and very satisfied responses decreased post-COVID (though satisfied responses decreased only slightly) while 
neutral, unsatisfied, and very unsatisfied responses increased (Figure 33). 


Course Interaction 
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A majority (66.1%) of students agreed or strongly agreed that “student-to-student and faculty-to-student 
interactions were encouraged and facilitated” and 66% indicated that peer interaction facilitated active learning 
through frequent and ongoing peer involvement. 


Administrators, staff, and faculty also overwhelmingly agree with the statement that “student-to-student and 
faculty-to-student interaction are essential characteristics and are encouraged and facilitated.” For 
administrators and staff, 80.4% and 84% of faculty agreed or strongly agreed. The data also indicates that 
interaction drives design with 76.3% of faculty agreeing or strongly agreeing that students participate in 
meaningful workgroups that promote collaboration among students, and 77.4% agreed or strongly agreed with 
the statement “peer interaction facilitates active learning through frequent and ongoing peer-involvement.” 
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The interview data brought more depth to the quantitative data, highlighting how poor technology choices, 
access, or design unintentionally impact student-to-student interactions: “Another problem is this interaction 
between students between each other...Any interaction is done with WhatsApp groups, emails, and things like 


that, and it’s much harder to find help between each other when all you have is a WhatsApp group that makes it 


much harder to gain knowledge from other.” 


Expert Recommendation: Establish the presence of the online educator. 


Target Faculty 
Audience 
Needs Opportunities for the student to get to know the teacher 


Initiatives 


Establish online presence of the educator 


Goals Provide opportunities for the student to better know their teacher 
Actions e Include an instructor biography and contact information 
e Share expertise with students through bringing outside resources (especially those 
written/prepared by the educator) 
e Regularly participate in discussions or other collaborative activities 
CSFs Increased engagement between the instructor and students 
KPIs e Instructor biography and contact information shared 


e Instructor regular contributing in the virtual classroom 


Subdimension: Feedback 


Measures of satisfaction with feedback both pre- and post-COVID, among both administrators/staff and faculty 
reflect that the majority of participants are overall satisfied or very satisfied. It is of note that both satisfied and 
very satisfied responses decreased post-COVID (though satisfied responses decreased only slightly) while neutral, 
unsatisfied, and very unsatisfied responses increased (Figure 34). 
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Both the interviews and survey highlighted the importance of detailed and timely feedback in online teaching and 
learning. Administrators and staff (82%) faculty and (86.7%) agreed or strongly agreed that faculty and instructors 
provide timely and detailed feedback on assessments and student inquiries. Students largely agree (37.5%) or 
strongly agree (28.4%) that they get ongoing meaningful feedback, but 17.9% disagreed or strongly disagreed. As 
with the section on engagement, the interview data provides additional insight but no solid answers. Interview 
themes seem to indicate that students appreciate clarity and timeliness, but the data is inconclusive ona 


consistent expectation for what that means. 


Recommendation: Set expectations for feedback provision and methods, as well as when 
feedback will be provided and how it is related to course learning objectives. 


Target Faculty 
Audience 
Needs Feedback provision and methods, timing, and connection to learning objectives is present and 


communicated to students 


Initiatives 


Develop and communicate methods for providing feedback, and set expectations for timing 
and explanations of how feedback is related to learning objectives 


Goals Support successful student learning outcomes by establishing and communicating clear 
processes and expectation for feedback 
Actions e Develop processes and methods of feedback provision in alignment with course 
activities 
e Communicate feedback processes and methods, and set expectations including 
timing and how feedback is connected to learning objectives, to students 
CSFs e Feedback processes and methods are developed and implemented 
e Communication to students regarding feedback processes, methods, connection to 
learning objectives, and expectations is provided 
KPIs e Processes and methods of feedback are present and implemented 


e Communication to students regarding feedback is present and implemented 
e Student evaluations indicate that feedback processes, methods, and provision are 
implemented, understood, and support learning 
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Recommendation: Implement student experience measurements within online learning 
experiences as a part of the design process. By asking students what quality means to 
them, iterative improvements can be made as befits learning outcomes. 


Target | Faculty 

Audience 

Needs Measure student experiences and quality perceptions as an integrated component of course 
design, and use the resulting data in continual improvement processes 

Initiatives Develop and implement measures of student experience and quality perceptions 

Goals To gain an understanding of student experiences and quality perceptions that are used in 
continual course improvement processes 

Actions Develop and implement measures of student experience and quality perceptions 

CSFs e Measures of student experience and quality perceptions are present and utilized 

e Resulting data is used in the continual improvement of courses 
KPIs e Measures of student experience and quality perceptions enhance understanding 


among administrators/staff and faculty regarding these items 
e Student experience and quality perception feedback results in course improvements 


addressing related course components 


Subdimension: Teaching & Learning Resources 


Measures of satisfaction with teaching and learning resources both pre- and post-COVID, among both 
administrators/staff and faculty reflect that the majority of participants are overall satisfied or very satisfied. It is 
of note that both satisfied and very satisfied responses decreased post-COVID (though satisfied responses 
decreased only slightly) while neutral, unsatisfied, and very unsatisfied responses increased (Figure 35). 


Resources 
50.0% 
40.0% 
30.0% 
20.0% 
, Zi 
Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied 
Admin/Staff/Faculty Satisfaction - Pre-Covid w Admin/Staff/Faculty Satisfaction - Post-Covid 
Figure 35 


I ONLINE LEARNING ™ 
O)E Cl CONSORTIUM 


The survey found interesting differences amongst administrators and staff, faculty, and students when it comes to 
the perceived richness that course resources bring to an online course. In response to the statement “course 
resources provided richness in learning materials and activities, support and instruction, instructor interactions, 
and tools and media,” 82.8% faculty and 78.6% of administrators and staff agreed or strongly agreed. Students, 
however, did not feel as strongly with only 64.3% agreeing or strongly agreeing and 19.4% disagreeing or strongly 
disagreeing (Figure 36). 
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The survey and interview data together indicate this trend could be explained by the differences in how faculty 
think their students experience courses versus how they feel they are experiencing them. These findings offer an 
opportunity for further research and discussion. 
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Recommendation: Develop and implement evaluation procedures to gather information 
regarding student perceptions of course resources, and include this data in continual 
course improvement processes. 


Target Administrators/Staff, Faculty 
Audience 
Needs An increased understanding of student perceptions and feedback regarding course resources 


Initiatives 


e Develop and implement evaluation procedures to gather student perceptions of 
course resources 
e Include this data in continual course improvement processes 


Goals Understand student perceptions regarding course resources for the purpose of improving 
courses to better meet student learning needs 
Actions e Develop and implement evaluation procedures regarding student perceptions of 
course resources 
e Utilize resulting data to inform continual course improvements 
CSFs e Evaluation procedures are developed and implemented 
e Evaluation results are utilized for continual course improvements 
KPIs e Evaluation processes result in student feedback on perceptions of course resources 


e Results are integrated into course improvement processes, which reflect inclusion of 


appropriate modifications based on identified needs 


Subdimension: Innovation 


Measures of satisfaction with innovation in online teaching and learning both pre- and post-COVID, among both 


administrators/staff and faculty reflect that the majority of participants are overall satisfied or very satisfied. It is 


of note that both satisfied and very satisfied responses decreased post-COVID while neutral responses increased 


and unsatisfied and very unsatisfied responses slightly increased (Figure 37). 
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Overall, administrators, staff, and faculty report satisfaction with innovation in online teaching and learning. At 
their institutions, administrators/staff (74.1%) and faculty (72.7%) agreed or strongly agreed that innovation is 
encouraged for this dimension. However, students report much lower agreement at 58.4% that their faculty 
regularly innovate their online teaching (Figure 38). Students might not be aware of all of the innovations that take 
place or administrators, staff, and faculty may not be aware of all of the new technologies and online learning 
approaches. 


Faculty and instructors are encouraged to innovate in 
their online teaching and in approaches to student 


learning. 
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Recommendation: Provide communications to increase student awareness of innovations 
implemented in online teaching and learning, and provide opportunities for student 
feedback regarding innovative practices. 


Target Students 
Audience 
Needs Provide students with information about innovations in online teaching and learning, and 


gather student feedback regarding the innovative practices 


Initiatives Provide communications and feedback opportunities to students regarding innovative online 
teaching and learning practices 


Goals Increase student awareness of innovative practices and incorporate their feedback as 
appropriate in continual improvement and innovation 


Actions e Increase student awareness of innovation in online teaching and learning 
e Develop and implement opportunities for student feedback on the success and 
development of innovative practices 


CSFs e Communications regarding innovation are provided to students 
e Student feedback opportunities are provided 
KPIs e Student feedback reflects an increased awareness of innovation in online teaching and 
learning 


e Student feedback is evaluated and incorporated into decision-making on innovative 


practices 


Dimension IV: Assessment 


The Assessment dimension measures five subdimensions including Assessment Strategies, Assessment Processes, 
Assessment Methods, Assessment Types, and Innovation. 


Prior to COVID, satisfaction measures in the assessment dimension indicated positive responses, through 
consistent decreases in satisfaction and increases in dissatisfied and neutral responses post-COVID as well as 
survey and interview data indicate that readiness for fully online teaching and learning was not met across all 
subdimensions. 


Overall findings in the Assessment dimension indicate that administrators/staff and faculty are consistently in 
alignment with regard to their opinions and beliefs across all subdimensions, suggesting that their interactions 
regarding assessment and the communication of assessment practices are effective among each other. In general, 
administrators/staff and faculty agree or strongly agree at rates of 70-85% that the queried practices in each 
subdimension are present. The most impactful findings in the Assessment dimension are illuminated when 
considering administrator/staff and faculty responses as compared to student responses on the same items, 
dimensions, and/or subdimensions. 


Like administrators/staff and faculty, student responses show consistent patterns of opinions and beliefs across 
all subdimensions. However, students’ agreement rates are primarily around 60%, while disagreement stands out 
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at approximately 20%. The consistency in response among administrators/staff and faculty as a group, as well as 
among students as a group lends strength to the primary finding that there are significant gaps in communication, 
expectations, and/or implementation of assessment strategies, processes, methodologies, and types of 
assessment between the group of administrators, faculty, and staff, and their students. These findings are 
described below by subdimension. 


Interview responses related to the assessment dimension overall supported the findings from the survey items on 
assessment. Selected interview responses are noted in the subdimension results below, though it is worth noting 
here that two themes emerged in the qualitative interviews that were not queried in the survey instrument. First, 
some faculty and administrators indicated concerns about academic integrity in online assessments - specifically, 
ensuring that the student enrolled was the individual completing assessments, and also that the assessments 
were accurately measuring the learning outcomes desired. Second, several faculty members and administrators 
noted during their interviews that they support online assessment overall, but noted that the rapid shift to remote 
instruction caused difficulties when assessments had been planned for other modalities and had to be moved 
online, and/or reported successful assessment experiences with theoretical knowledge, but difficulty designing 
and implementing assessments for practical fields. Regarding academic integrity, professional development on 
the use of technological options to ensure appropriate behavior is recommended, as well as professional 
development on designing and delivering assessments in ways that intrinsically encourage academic honesty. 
Professional development should also be considered for the development and delivery of assessments that are in 
alignment with measuring the desired objectives. Finally, regarding the challenge incurred with practical subject 
matter, an understanding of areas in which this is a concern is needed, followed by approaches of innovative 
online assessment techniques to address these challenges, the discovery and use of alternative solutions such as 
simulations, home labs, etc., and the consideration of hybrid or other modalities that may be employed to bridge 
any remaining gaps in assessment for practical subject matter. 


Subdimension: Assessment Strategies 


Measures of satisfaction with assessment strategies both pre- and post-COVID, among both administrators/staff 
and faculty reflect that the majority of participants are overall satisfied or very satisfied. It is of note that both 
satisfied and very satisfied responses decreased post-COVID (though satisfied responses decreased only slightly) 
while neutral, unsatisfied, and very unsatisfied responses increased (Figure 39). 
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Assessment strategies in this study encompassed student readiness for online learning, piloting strategies that 
include digital and other pertinent literacies, and formative assessment. Administrators/staff, faculty, and 
students all responded in alignment with the consistent pattern noted above. Qualitative interview responses 
further supported these findings, and respondents provided discussion not only of readiness, particularly in the 
remote instruction environment, but also gave clear examples of piloted assessment strategies to address 
learning needs, as well as noted a diverse set of formative assessment strategies. 


With regard to the assessment strategies subdimension, the primary recommendations are to address the gap in 
administrator/staff and faculty, and student perceptions include increased presence of clear communication to 
students identifying assessments, their purposes, and the learning outcomes that are being assessed. 


Recommendation: Develop and implement course-level communications to students that 
identify assessments, their purposes, and the learning outcomes for each. 


Target Faculty, Students 

Audience 

Needs Clearly identify assessments and identify their purposes and learning outcomes to students 

Initiatives Develop and implement course-level communications to students regarding assessments, their 
purposes, and the learning outcomes addressed by each 

Goals Increase student understanding of assessment practices in order to support successful student 
learning outcomes 

Actions Faculty provide communications to students via various methods (syllabi, email, LMS 
communications, etc.) identifying assessments, their purposes, and the learning objectives for 
each 

CSFs Faculty provide communications to students regarding assessments 

KPIs e Communications are provided to students regarding assessments 


e Student evaluations and feedback indicate increased awareness and understanding of 
assessments, their purposes, and the associated learning outcomes 
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Subdimension: Assessment Processes 


Measures of satisfaction with assessment processes both pre- and post-COVID, among both administrators/staff 
and faculty reflect that the majority of participants are overall satisfied or very satisfied. It is of note that both 
satisfied and very satisfied responses decreased post-COVID while neutral, unsatisfied, and very unsatisfied 
responses increased (Figure 40). 
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Assessment processes in this study included the alignment of learning objectives and assessments, the effective 
communication of assessment criteria, grading, and expectations, and feedback. Findings in this subdimension 
followed the clear pattern of responses noted above, with the majority of administrators/staff and faculty 
indicating the each of these items is present at high rates, with an approximate 20 percentage-point difference in 
student responses (~60% agreement) and 20% of students disagreeing that these elements were present and/or 
effective (Figures 41 and 42). Qualitative interviews did indicate faculty awareness of difficulties with 
implementation and potentially student understanding of assessment processes due to the pandemic. 
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Assessment criteria are clearly described when the 
assignment is issued with grading rubrics and samples of 
assignments illustrating instructor’s expectations are 


provided. 
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Frequent and substantial feedback from the instructor is 
provided to students through assessments. 
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Figure 42 


Recommendations for the assessment processes subdimension include further ensuring alignment of learning 
objectives and assessments through quality review processes, as well as to implement consistent course design 
that is aligned with best practices for the logical placement and effective use of communications and materials 
regarding criteria, assignments, expectations, and grading rubrics. Feedback provision and methods, as well as 
explicit expectation setting for when feedback will be provided and how feedback is connected to each 
assessment may also be useful in student understanding and benefit resulting from these practices. 
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Recommendation: Ensure alignment of learning objectives and assessments through 
quality review processes. 


Target Faculty 

Audience 

Needs Ensure alignment of learning objectives and assessments 

Initiatives Ensure alignment of learning objectives and assessments through quality course review 
processes 

Goals Support successful student learning through alignment of learning objectives and 
assessments 

Actions Include alignment of learning objectives and assessments in quality course reviews 

CSFs Learning objectives and assessments are aligned 

KPIs Successful student outcomes increase as a result of effectively aligned learning objectives 
and assessments 


Recommendation: Align course design practices regarding assessment with best practices 
for the placement and use of communications and materials including criteria, 
assignments, expectations, and grading rubrics. 


Target Faculty 

Audience 

Needs e Implement course design best practices regarding assessment, including delivery and 
placement 


e Aligncommunications and materials, including criteria, assignments, expectations, 
and grading rubrics with best practices 


Initiatives Ensure that assessments and related communications in online courses follow best practices 
Goals Support successful student learning outcomes through the use of effective course design and 
communication practices related to assessment 
Actions e Align course assessments with effective course design, including delivery and 
placement 


e Ensure that criteria, assignments, expectations, and grading rubrics are present and 
that these elements are clearly communicated to students 


CSFs e Course assessment delivery and placement follow course design standards 
e Criteria, assignments, expectations, and grading rubrics are present and 
communicated to students 


KPIs e Effective delivery and placement of assessments is evident in quality course reviews 

e Communications regarding criteria, assignments, expectations, and grading rubrics 
are provided to students 

e Student evaluations and feedback reflect presence and understanding of criteria, 
assignments, expectations, and grading rubrics 


62 es 


Recommendation: Set expectations for feedback provision and methods, as well as when 
feedback will be provided and how it is connected to each assessment. 


Target Faculty 
Audience 
Needs Feedback provision and methods, timing, and connection to assessments is present and 


communicated to students 


Initiatives 


Develop and communicate methods for providing feedback, and set expectations for timing 


and explanations of how feedback is connected to each assessment 


Goals Support successful student learning outcomes by establishing and communicating clear 
processes and expectation for feedback 
Actions e Develop processes and methods of feedback provision for each assessment 
e Communicate feedback processes and methods, and set expectations including 
timing and how feedback is connected to assessments, to students 
CSFs e Feedback processes and methods are developed and implemented 
e Communication to students regarding feedback processes, methods, connection to 
assessment, and expectations is provided 
KPIs e Processes and methods of feedback for assessments are present and implemented 


Communication to students regarding feedback is present and implemented 
Student evaluations indicate that feedback processes, methods, and provision are 
implemented, understood, and support learning 


Subdimension: Assessment Methods 


Measures of satisfaction with assessment methods both pre- and post-COVID, among both administrators/staff 


and faculty reflect that the majority of participants are overall satisfied or very satisfied. It is of note that both 


satisfied and very satisfied responses decreased post-COVID while neutral, unsatisfied, and very unsatisfied 


responses increased (Figure 43). 


Figure 43 
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The majority of survey questions directly measuring the assessment methods subdimension were directed to 
students. In a continuation of the response pattern indicated throughout the assessment dimension student 
responses, each of the items in this subdimension were rated as agree or strongly agree by approximately 65% of 
students, and approximately 20% disagreed or strongly disagreed with each. Items measured included the use of 
new assessment methods, the variety of assessments provided, the use of multiple and timely assessment 
activities, and finally, whether students believed that their overall course grades were not based solely on exams 
and quizzes. Qualitative interview responses indicated that while many faculty do focus on assessment methods 
such as midterms and finals, many are tuned into a rich variety of assessments spanning assessment 
methodologies, modalities, and diverse ways of demonstrating learning. 


Additionally in this subdimension, faculty and students were asked to respond to survey items regarding self- 
assessment, peer review, and feedback opportunities. Interestingly, faculty and students were more closely 
aligned regarding peer assessment than on any other survey questions on assessment. This is supported by 
qualitative interview data, with several respondents noting their efforts for collaboration, teamwork, and peer 
assessments. However, a gap emerged when students were about opportunities to self-assess their progress, 
opportunities for improvement, and set goals for personal and professional growth. Those who disagreed or 
strongly disagreed did so in alignment with other assessment items, at 21.8%. When considering agreement or 
strong agreement, only 41.4% of students responded accordingly, indicating not only a gap on this particular item, 
but a significant portion of students (36.7%) did not have a sense of either agreement or disagreement (Figures 43, 
44, and 45). 


Students’ self-assessments and peer review and feedback 
opportunities exist throughout the course. 
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Students’ self-assessments and peer review and feedback 
opportunities exist throughout the course. 
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Students had opportunities to learn from their peers 
through peer evaluation of one another’s work and other 


activities. 
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Figure 46 


Recommendations addressing the assessment methods subdimension include, as previously, clear 
communication of assessment opportunities, purposes, expectations, and how the factor into the final grade. 
Additionally, faculty may benefit from professional development on the design and delivery of both peer- and self- 
assessment methods in order to maximize their use and value, and to appropriately communicate and provide 
feedback to students on these items. 
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Recommendation: Provide faculty with professional development opportunities on the 
design and delivery of both peer- and self-assessment methods for students. 


Target Faculty 
Audience 
Needs Provide professional development opportunities on the design and delivery of peer- and self- 


assessment methods for students 


Initiatives Develop and deliver professional development opportunities for faculty on the design and 
delivery of peer- and self-assessment methods for students 


Goals Increase the effective use of peer- and self-assessments in courses for the purpose of 
supporting student learning 


Actions Develop and deliver professional development opportunities for faculty on the design and 
delivery of peer- and self-assessment methods for students 


CSFs Professional development opportunities are developed and provided to faculty on peer- and 
self-assessment methods for students 


KPIs e Faculty participate in professional development opportunities 
e Faculty develop and implement peer- and/or self-assessment opportunities for 


students 


Subdimension: Assessment Types 


Measures of satisfaction with assessment types both pre- and post-COVID, among both administrators/staff and 
faculty reflect that the majority of participants are overall satisfied or very satisfied. It is of note that both satisfied 
and very satisfied responses decreased post-COVID (though satisfied responses decreased only slightly) while 
neutral, unsatisfied, and very unsatisfied responses increased (Figure 47). 
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Administrators/staff, faculty, and students were all asked to rate their level of agreement or disagreement with the 
assertion that “opportunities are present to demonstrate proficiency in ways other than just tests and quizzes.” 
Response patterns by group were in alignment with the other items in the assessment dimension. Again, interview 
data support these findings, and several examples of varied assessment types are included in their comments. 
Recommendations for this subdimension include increased volume and clarity of communication to students 
regarding what is considered assessment, the purpose of each assessment, and expectations and grading for each 
assessment, which are covered in the detailed recommendations from previous Assessment subdimensions. 


Expert Recommendation: Provide opportunities to assess learning at higher levels of 
knowledge acquisition. 


Target Faculty 

Audience 

Needs Assessments for higher order thinking 

Initiatives Ensure knowledge acquisition 

Goals Implement assessments to reflect higher order thinking 

Actions Develop assessments (e.g., papers, projects, group activities, etc.) that reflect knowledge 
acquisition at a higher level 

CSFs Learners acquire needed knowledge 

KPIs Courses include assessment activities that fit higher order thinking on Bloom’s Taxonomy 


Subdimension: Innovation 


Measures of satisfaction with assessment innovation both pre- and post-COVID, among both administrators/staff 
and faculty reflect that the majority of participants are overall satisfied or very satisfied. It is of note that both 
satisfied and very satisfied responses decreased post-COVID while neutral, unsatisfied, and very unsatisfied 
responses increased (Figure 48). 
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The innovation subdimension for assessment was measured indirectly by a number of the questions described in 
the previous subdimension findings. Innovation may be measured through queries on new strategies and the 
inclusion of current and relevant literacies, the use of new methods, and opportunities to demonstrate learning in 
a variety of ways. Qualitative data indicate high interest in innovative assessments for online courses, both in 
current practice as well as exploring new opportunities. 


The recommendations above are relevant to innovation, as well as direct attention to the discovery and 
dissemination of innovative assessment strategies, processes, methods, and types. Further, faculty and others 
should be directly encouraged to include innovative assessments in their online courses, and provided training on 
design, development, and delivery of these innovations. 


Recommendation: Innovation in assessment strategies, processes, methods, and types is 
encouraged and supported, and related professional development on design, 
development, and delivery is provided. 


Target Faculty 
Audience 
Needs e Encourage the discovery, dissemination, and implementation of innovative 


assessment strategies, processes, methods, and types 
e Provide professional development on the design, development, and delivery of 
innovative assessment 


Initiatives e Explicitly encourage innovative assessment through communications, activities, and 
professional development 
e Develop and implement professional development opportunities on innovative 


assessment 
Goals Increase the design, development, and delivery of innovative assessment 
Actions e Develop and implement strategies for the discovery and dissemination of innovative 


assessment techniques 
e Develop and implement professional development opportunities for faculty on the 
design, development, and implementation of innovative assessment 


CSFs e Innovation relevant to assessment is actively encouraged 
e Professional development opportunities related to innovative assessment are 
provided 
KPIs e Clear encouragement of innovative assessment is communicated to faculty 


e Faculty participate in professional development opportunities related to assessment 


e Faculty implement and measure innovative assessment practices in their courses 


68 SSS aS 


Dimension V: Technology 


The Technology dimension measures nine subdimensions including Centralized Online Education Infrastructure, 
Coverage, Innovation, Internet Access, Information Technology Service Management (ITSM) Compliance, 
Modality, Operability, Reliability, and Security. 


Prior to COVID, satisfaction measures in the technology dimension indicate overall congruence among 
administrators, staff, and faculty, and this largely remained following COVID. As supported by survey and interview 
data, readiness needs were primarily focused on aspects of internet access, coverage, and reliability. Results 
suggest that improvements were needed pre-COVID on internet access, coverage, and reliability, and these areas 
were not sufficient to support fully-online course delivery on such a large scale. 


Overall findings across administrators and staff, faculty and student groups all indicate generally positive 
technology and perceptions across all nine subdimensions. Administrators and staff tended to have slightly 
different perceptions of the state of technology strategy and implementation than did faculty and students. 
Patterns in the qualitative responses indicate this largely due to the differentiation in campus roles; where 
administrators and staff are the ones in operational technology roles, faculty and students are in the roles of 
technology customer, thus not as knowledgeable of the technology landscape at the operational strategy level. 


Subdimension: Centralized Infrastructure 


Measures of satisfaction with centralized infrastructure both pre- and post-COVID, among both 
administrators/staff and faculty reflect that the majority of participants are overall satisfied or very satisfied. It is 
of note that post-COVID, satisfied responses decreased but very satisfied responses increased (Figure 49). 


Centralized Online Education Infrastructure 
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Administrators and staff noted that centralized support infrastructure was made widely available, including 
support for building and maintaining the technical infrastructure with 46.1% saying they agreed and 27.8% saying 
they strongly agreed. 
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The qualitative data is supportive in the sense of the availability and variety of technical offerings with the caveat 
of capacity. “We have invested a lot in our technology, in our network, high speed network, on the equipment, the 
pieces, the headphones, the cameras, the latest technologies... everything is connected together” says one 
respondent, while another says “Our real problem is we struggle with the infrastructure a little bit because we 
didn’t expect that huge usage of Internet in the ecampus. Or in our server.” 


Recommendation: Evaluate technology infrastructure to ensure that it is sufficient to 
meet current and upcoming demands, and create and implement improvement plans as 
appropriate. 


Target Administrators, Staff 
Audience 
Needs Evaluate technology infrastructure to ensure that it is sufficient to meet demand; create and 


implement improvement plans as appropriate 


Initiatives e Evaluate technology infrastructure 
e Utilize evaluation results to create and implement any necessary improvement plans 


Goals Ensure that technology infrastructure is sufficient to meet current and anticipated demands, 
increasing the likelihood of smooth online learning experiences for faculty and students 


Actions e Evaluate technology infrastructure 
e Utilize evaluation results to create and implement improvement plans as needed 


CSFs e Technology infrastructure is evaluated for ability to meet demand 
e Results are utilized to create and implement any necessary plans for improvement 


KPIs e Technology capabilities and capacities are understood and needs identified 
e Technology infrastructure is improved as needed to support demand 


e Technology infrastructure adequately meets the needs of faculty and students 


Subdimensions: Internet Access, Reliability 


Measures of satisfaction with internet access both pre- and post-COVID, among both administrators/staff and 
faculty remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied 
(Figure 50). 
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Internet Access 
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Measures of satisfaction with reliability both pre- and post-COVID, among both administrators/staff and faculty 
remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied (Figure 
51). 
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The subdimensions of Internet access and reliability are closely linked as they were frequently referenced together 
in the qualitative data. While overall technology reliability was seen as relatively high across all respondents, the 
same was not found for Internet reliability. Internet access was viewed as critical to success across all of the 
respondent categories, but answers were mixed as to whether access was available in widespread, reliable ways. 
While 66.9% of students reported that they either agreed or strongly agreed that they have internet available 
anytime to work online, only 51.5% agreed or strongly agreed that the institution provided reliable access to 
wireless Internet (Figure 52). This contrasts with the and 69.8% and 72.3% of administrative staff and faculty 
respectively say the institution provides reliable access to wireless internet, the qualitative data tells a slightly 
different story. Internet access on campus may be widely available, but it is unclear as to whether students have 
reliable access at a distance. “Some are from far places or some villages, you know, not actually the city’s 
students... They find some difficulties ... joining the lecture all the time. They are going out and joining in again 
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because of the Internet,” said one faculty member. “Our real problem,” said another, “is that most, some of them, 
they have weak computers, weak connection. So that’s the challenging part of teaching online.” 


The institution provided reliable access to wireless 
internet (WiFi) that is available across the campus, 
including outdoor spaces. 
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Recommendation: Given challenges with identifying whether Internet access will be 
reliable, or where students will be located when completing their online courses, 
alternatives should be considered such as a reduction in the number of synchronous 
course meetings, or technology and faculty development working together to support 
faculty in offering different, flexible access options that do not rely heavily on 
immediately accessible technology. 


Target Administrators, Staff, Faculty 
Audience 
Needs Flexible options for completing online courses to account for internet reliability and/or 


student needs 


Initiatives e Develop and implement flexible course participation and completion options 
e Collaborations between faculty and technology administrators/staff to support 
flexible course options 


Goals Improve the ability of faculty and students to successfully engage in online education 
Actions Develop, implement, and support flexible options for course participation and completion 
CSFs Presence and awareness of flexible course participation and completion options 

KPIs e Faculty and student use of flexible options to support online learning 


e Evaluations and feedback indicate efficacy and result in recommendations for 
potential improvements 
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Asecondary recommendation that emerged repeatedly in the qualitative data was the idea of using university 
funds to procure and distribute needed equipment such as functional laptops to students and faculty for the 
purposes of support in not only the ability to access learning materials, but in ease of support for administrators 
and staff who would then maintain a consistent fleet of technology across the board. While this recommendation 
is not directly related to modality, it is related to the primary recommendation for this subdimension (above) and 
bears consideration for effective faculty and student support. 


Subdimension: Information Technology Service 
Management (ITSM) Compliance 


Measures of satisfaction with ITSM compliance both pre- and post-COVID, among both administrators/staff and 
faculty remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied 
(Figure 53). 
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Administrators and staff noted strongly that an IT Service Management (ITSM) plan was in place to design, plan, 
deliver, operate, and control information technology in support of online program development; 46.6% agreed 
while 30% strongly agreed. In the qualitative data, service lines were evident in regard to technology support such 
as server and desktop help, but less evident in academic technology or instructional design support. In regard to 
planning support, one respondent said “we were only talking about the infrastructure level of the IT technology, 
like assisting our labs, increasing number of laptops” while another noted difficulties “dealing with the support of 
instructors and they’re still figuring out the Blackboard and the online teaching aspect for the lecture based 
courses moving online.” 
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Recommendation: Create faculty development and support materials around how the 
academic technologies offered by IT work, and incorporate these services purposely into 
overall ITSM structures. 


Target Faculty 

Audience 

Needs Faculty development and support materials on the academic technologies supported by IT 

Initiatives Develop and disseminate faculty development and support materials related to academic 
technologies’ use and available support through IT 

Goals To ensure awareness and support for academic technologies, including instructional design 
support and just-in-time technical resources 

Actions Develop and disseminate faculty development and support materials related to academic 
technologies supported by IT 

CSFs e Faculty development and support materials related to academic technologies are 

developed 


e Communications are provided regarding these resources to increase faculty 
awareness and use of academic technologies and associated support needs 


KPIs e Faculty development and support materials related to academic technologies are 
developed, disseminated, and accessed by faculty 
e Increased use of academic technologies and/or support resources 


Subdimension: Modality 


Measures of satisfaction with modality both pre- and post-COVID, among both administrators/staff and faculty 
remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied (Figure 
54). 
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Students were generally in agreement that online courses were delivered with an effective mix of synchronous or 
asynchronous delivery, depending on the learning outcomes. While 18.9% were neutral about the mix, 38.1% of 
students were in agreement and 28.8% strongly agreed that they experienced both modalities depending on the 
delivery needs of their learning objectives. This is in strong contrast with faculty who report at 53.2% agreement 
and 27.7% strongly in agreement. Administrators and staff strongly agreed more at 29.2%, but tended to be 
between faculty and student ratings (Figure 55). 


The learning management system enables online courses 
to be delivered asynchronously and synchronously, 
dependent on the learning objectives. 
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Recommendation: Evaluate student experiences, needs, and perceptions regarding 
synchronous and asynchronous learning in online courses and incorporate findings into 
course design and delivery. 


Target Students 
Audience 
Needs Understand student experiences, needs, and perceptions regarding synchronous and 
asynchronous learning, and utilize those findings to inform course design and delivery decisions 
Initiatives e Evaluate student experiences, needs, and perceptions regarding synchronous and 
asynchronous learning in online courses 
e Include findings in course design and delivery decision-making 
Goals Support successful student learning experiences through the evaluation of synchronous and 
asynchronous experiences, needs, and perceptions and implementation of effective course 
design and delivery to support student needs 
Actions e Gather student feedback on their experiences, needs, and perceptions regarding 
synchronous and asynchronous online learning 
e Evaluate feedback and incorporate findings in decision-making processes for online 
course design and delivery 
CSFs e Student feedback is collected 
e Feedback is evaluated for student needs 
e Needs are addressed through course design and delivery 
KPIs e Student feedback regarding experiences, needs, and perceptions of synchronous and 


asynchronous online learning is regularly collected, evaluated, and incorporated into 
course design and delivery 
e Student feedback and course outcomes indicate effective course design and delivery 


Subdimensions: Operability, Security 


Measures of satisfaction with operability both pre- and post-COVID, among both administrators/staff and faculty 
remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied (Figure 


56). 
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Measures of satisfaction with security both pre- and post-COVID, among both administrators/staff and faculty 
remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied (Figure 
57). 
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Subdimensions operability and security work closely in the support of online education. When asked if the 
learning management system, the primary delivery mechanism for online courses, had an established 
interoperability protocol established so that embedded applications could provide a seamless experience, 70.3% 
of administrators and staff either agreed or strongly agreed. In regard to security, 76% of administrators and staff 
either agreed or strongly agreed that digital security measures were in place to ensure the integrity and validity of 
information including identity access. 
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Recommendation: Operability and security should be regularly monitored and 
functionality maintained and updated as needed. 


Target Administrators, Staff 

Audience 

Needs Regular monitoring 

Initiatives Operable and secure technology to support online learning 

Goals Ensure the regular functioning and security of technology to support online learning 

Actions Establish protocols for regular review of the learning management system and other 
technologies to ensure they continue to operate as required and remain secure 

CSFs Functioning technology to support online learning 

KPIs e No, or limited, downtime for the learning management system or other technologies 


e Nosecurity breaches with the learning management system or other technologies 


Subdimension: Technology Plan 


Administrators and staff are varied in their answers about whether there is a documented technology strategic 
plan in place to ensure quality standards in online learning. While 45.7% agreed and 25.8% strongly agreed, 18.7% 
reported being neutral in their perceptions. Patterns in the qualitative data indicate that while the strategic 
technology plan may be known, its connection to online learning may not always be as evident. That said, the 
clearest connection in the qualitative data between the strategic technology plan and the success of online 
learning was in instructional technology support such as faculty development. “We are moving to more often 
giving trainings on instruction of technology and also how the courses could be standardized,” said one 
respondent. “So far we have finished a plan of how to create and produce courses. We’re trying also to provide 
more resources for how tools could be used for that purpose.” 
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Recommendation: Administrators/staff and faculty receive communications regarding 
technology plans, including clear indicators of how the plan is connected to, and 
supports, online education. 


Target Administrators/staff and Faculty 

Audience 

Needs Increase awareness of technology plans and how they connect with and support online 
education 

Initiatives Develop and implement a communication plan for administrators/staff and faculty regarding 
technology plans and online education 

Goals Increase awareness and understanding of technology plans, particularly as related to online 
education in order to better support stakeholder needs 

Actions Develop and implement a communication plan for administrators/staff and faculty regarding 
technology plans and online education 

CSFs Communications are provided to administrators/staff and faculty regarding technology plans 


and online education 
KPIs Administrators/staff and faculty report increased awareness of technology plans and how they 
connect with and support online education 


Subdimension: Coverage 


Measures of satisfaction with coverage both pre- and post-COVID, among both administrators/staff and faculty 
remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied (Figure 
58). 
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Administrators, staff, and faculty mostly felt that their institution had sufficient coverage and were generally 
satisfied pre- and post-COVID-19. Administrators/staff reported at 71.9% and faculty at 75.1% that the technology 
used supports students as they access and move through an online course (Figure 59). 
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Course embedded technology actively supports the 
achievement of learning outcomes and delivery of course 
content, and superfluous use of technology is minimized. 
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Students reported far greater extremes for coverage with stronger disagreement, but also stronger agreement. 
However, 73.1% of students agreed or strongly agreed that they were able to access all that they needed for online 
learning in one place (Figure 60). 


The course sites included embedded technologies and 
functionality that supported learning. 
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Recommendation: Continual monitoring of coverage and access to online learning needs 
for administrators/staff, faculty, and students. 


Target Administrators, staff, faculty, and students 
Audience 
Needs Regular monitoring 
Initiatives Continuous coverage and access to online learning technology 
Goals Ensure that administrators, staff, faculty, and students are able to access technology needed 
for online courses 
Actions Establish programs for review of technologies to make sure access is maintained 
CSFs Technologies available to administrators, staff, faculty, and students 
KPIs Coverage and access to online learning technologies maintained 


Subdimension: Innovation 


Measures of satisfaction with technology innovation both pre- and post-COVID, among both administrators/staff 
and faculty indicate that the majority of participants are overall satisfied or very satisfied. Post-COVID, satisfied 
responses decreased slightly and neutral responses increased (Figure 61). 
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Overall, administrators, staff, and faculty report that they are satisfied with their institution’s online learning 
innovation. Administrators/staff (73.3%) and faculty (72.4%) report that their institution pilots, supports, and 
encourages innovative technologies; however, students report lower at 64.4% (Figure 62). 
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New and innovative technologies are piloted, supported, 
and encouraged in online courses on a timely basis at the 


institution. 
50.0% 
40.0% 
30.0% 
20.0% 
0.0% _ 
Strongly Disagree Disagree Neutral Agree Strongly Agree 
Admin/Staff mFaculty Students 
Figure 62 


Recommendation: Provide opportunities for student feedback regarding innovative 
technology. 


Target Students 

Audience 

Needs Include student feedback regarding the use and implementation of innovative technology 

Initiatives Provide opportunities for students to give feedback on implemented tools/practices, and for 
them to share their thoughts, needs, and ideas on innovation 

Goals Increase student awareness of innovative practices and incorporate their feedback as 
appropriate in continual improvement of technology innovation 

Actions Develop and implement opportunities for student feedback on the success and development 
of innovative practices in technology 

CSFs Student feedback opportunities are provided 

KPIs Student feedback is evaluated and incorporated into decision-making on innovative practices 


Dimension VI: Student Support 


The Student Support dimension measures five subdimensions including Student Orientation, Equity, Accessibility, 
Compliance Standards, and Innovation. 


Prior to COVID, satisfaction measures in the student support dimension indicated positive responses, through 


survey and interview data indicate that readiness for supporting students in a fully online environment needed 
improvement across all subdimensions, with the exception of compliance standards. As detailed below, this 
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dimension revealed interesting differences with regard to perception of readiness between private and public 
university respondents. 


The findings in the Student Support dimension indicate a discrepancy between the students’ experiences and the 
administrators/staff and faculty perceptions of the availability of student support for their online courses provided 
when the global pandemic prompted a rapid, worldwide adoption of remote learning. The disparity between the 
groups’ opinions remains consistent within each of the five subdimensions with exception of Compliance 
Standards. All three stakeholder groups reported similar, positive responses regarding institutional adherence to 
policies and agreements. Administrators/staff and faculty perceptions remain aligned across all subdimensions, 
suggesting positive communication strategies and effective interactions regarding student support. While the 
students’ responses remained constant in their relation to the administrators/staff and faculty opinions, they vary 
significantly by the students’ institution type (public or private). The interview data provides a broader picture of 
the variability in institutional approaches to repositioning all courses to the digital environment. 


As displayed in the graphs at the start of each subdimension, there was very little change in the 
administrators/staff and faculty pre- and post-Covid-19 satisfaction with student support subdimensions. This 
data point is perplexing when juxtaposed with students’ opinions of institutional readiness for online learning and 
deserves further analysis. 


The interview format provided the students, administrators/staff, and faculty an opportunity to share 
unstructured responses that afforded insight into the unanticipated obstacles many students had to overcome 
during their online learning. Learning from home presented many unexpected challenges for both students and 
faculty. A public university faculty honed in on the unforeseen chaos of shared learning: “You’re doing it in the 
comfort of your own home. But at the same time, everybody is at home. And so you’re kind of trying to adapt your 
schedule to everyone that is learning with you and using platforms, especially if you’re a mom. You have to make 
sure your children are opening their laptops and doing that online, learning as well. And | think it’s, you know, 
overwhelming for the students as well.” Several students noted the stress, anxiety, and mental confusion 
stimulated by the appearance of a rare global pandemic. One public university student worded the experience 
very succinctly: “I think we were all mentally unstable.” The student concluded: “So | think it affected our grades 
and our learning.” 


As noted, the students’ experiences varied by institution type. The data suggests that many students enrolled in 
private universities were unfamiliar with online learning, and this inexperience informed their beliefs about their 
institution's preparedness for distance education. Public university students had a more positive perspective on 
their institutions’ readiness for the rapid conversion. While almost 40% of students attending public universities 
found the COVID-19 transition to be relatively smooth, none of the students attending private universities 
reported a seamless transition. 


Subdimension: Orientation and Support 


Measures of satisfaction with orientation and support both pre- and post-COVID, among both administrators/staff 
and faculty indicate that the majority of participants are overall satisfied or very satisfied. Post-COVID, very 
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satisfied responses decreased, neutral responses increased and there were slight increases in unsatisfied and very 
unsatisfied responses (Figure 63). 


Student Orientation & Support 
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Early and ongoing orientation and training opportunities can set clear expectations and accelerate the learning in 
digital education spaces. During the interview, one private university student posited his thoughts on the need for 
an orientation: “Maybe it was because it was our first time to use the online apps and programs. We never had 
online courses before, so we didn’t know how to use it. But after they announced that we had to take the online 
courses, they sent us an email with instructions on how to use it....” A faculty member at a public institution 
reflected on the need for clearly stated, upfront expectations: “So | think once everybody understands what’s 
expected or how they’re expected to learn online, then | think it makes it a little bit easier to determine basic 
methods of what we need to do.” 


Students, administrators/staff, and faculty were asked to rate their agreement with the availability of several 
student support services (Figure 64). Administrators/staff, faculty, and students agreed that institutions provided 
them with an orientation to the course design and technologies, (81%, 84%, and 72%, respectively), information 
on university technology and academic support resources (82%, 84%, and 68%, respectively), and access to 
personnel who could assist them with their technology and academic support needs (81%, 80%, 67%, 
respectively). While approximately two-thirds of students agreed that the services were made available to them, 
Figures 64 and 65 display the students’ reported usage and helpfulness of the resources. More than 90% of 
students made use of the program technology help desk and other faculty when seeking assistance, and the vast 
majority noted some degree of satisfaction with the service they received. The data indicates that students relied 
upon friends, not classmates, during this time, which could suggest a need for emotional and social support. 
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Availability of Student Support Resources 
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Supports Used by Students 
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A caveat, the students did not report the nature of the help they requested from each source. The interviews 
supplement this data by providing more context to the problems students hoped to resolve. Many faculty and staff 
noted the rapid transition led to an increased need for them to provide assistance to students outside of the 
course content. For example, a staff member from a private university commented: “Our students are e-mailing us 
and chatting with us about other issues not related to e-learning. Sometimes they ask us about registration and 
admin and admission... Sometimes they ask us about their concerns regarding tuition or payment or any other 
issues. So | believe they choose us over the department because we respond quickly and immediately and we 
provide feedback that they believe or they can rely on.” Several students noted the delay in response time to 
requests that required immediate resolution including one private university student: “...they couldn’t respond to 
us quickly because there were a lot of questions from students.” A staff member at a public university buttressed 
the student’s concerns: “The pace of courses required quick response.” 


Helpfulness of Student Support Resources 


60% 
50% 
40% 
30% 
20% 
10% 

0% 


mNotatAll wSlightly/Moderately mVery/Extremely 


Figure 66 


86 SaaS SS 


Recommendation: Ensure availability of orientations and student resources targeted to 
support students and faculty in online learning environments. 


Target Faculty, Students, Advisors, Information Technology Staff, Instructional Designers 
Audience 
Needs e Reduce response time to address student problems/complaints 
e Increased availability of students support services to address the specific needs of 
students in online learning environments 
Initiatives Ensure technology training, tools/hardware, and resources to support best practices and 
academic success are available to all faculty and students engaged in online learning 
Goals Enrich and support the student and faculty online learning experience to ensure successful 
student outcomes 
Actions e Extend the breadth and depth of orientation topics 
e Introduce Online Learning Readiness assessment prior to enrollment in online courses 
e Provide voluntary, interactive training opportunities 
e Invest in technology/staffing 
© toscale, streamline student problem/complaint mechanisms and reduce 
response time; 
© to deliver online resources comparable to those available to on-campus 
students; and 
©  toensure all faculty and students in online courses have required technology 
tools 
CSFs e The technology and teaching and learning requirements for online courses is clearly 
communicated to faculty and students 
e The availability of student support resources is clearly communicated to faculty, 
advisors, and students 
KPIs e Students’ surveys and feedback report prompt responses to problems/complaints 


e Students’ surveys and feedback report the availability and helpfulness of support 
services 


Expert Recommendation: Provide student support equal to those provided to face-to-face 


students. 


Target Audience Administrators, staff 


Needs Student support for online learners 
Initiatives Provide student services to online learners that is equivalent to services provided in-person 
Goals Provide services through virtual modality for students unable to access in-person 
Actions Review existing student services and provide options for online students to access 
CSFs Improved service to online students 
_ KPIs Online students able to access needed services through virtual modality 
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Subdimension: Equity 


Measures of satisfaction with equity both pre- and post-COVID, among both administrators/staff and faculty 
indicate that the majority of participants are overall satisfied or very satisfied. Post-COVID, satisfied and very 
satisfied responses decreased and neutral, unsatisfied and very unsatisfied responses increased (Figure 67). 
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All students, including underrepresented students, need the ability to access the resources (e.g., technology, 
course materials, support) to achieve academic success. According to the survey data, the majority of 
administrators/staff (78%), faculty (78%), and students (66%) believe these were available. However, all groups 
were less certain about the availability of tools and other resources to assist students challenged by disabilities. 
Administrators/Staff agreed or strongly agreed at 70%, with faculty at 65%, and students at 56%. 


The interview data provided insight into the obstacles created by limited technology resources and/or access. 
Regardless of the challenges, several faculty and administrators/staff noted examples of diversified learning 
opportunities that they provided to students to increase interest, engagement, and learning. A public university 
faculty member shared several positive practices: “The discussion boards provided an opportunity for students 
who weren’t able to have a good Internet connection or ability to get online at that moment or stay online 
because some of them would be there for a minute and then just disappear. It gives them an opportunity to still do 
the work... And again, being flexible and having some sort of collaborative environments for them to work 
together because they still need the student to student interaction as well.” 


Faculty at both private and public institutions shared the importance of flexibility, communication, patience, and 
humor to maintain a positive, equitable learning environment. From a public university educator: “Providing 
maybe more asynchronous options might be a good idea, especially for those like | said, who either don’t have 
good Internet access or have other siblings who are online or parents who are working online at the same time 
and then having to log off and on so often.” And another public university faculty shared: “It was very important 
for me to be very flexible with the student who could not get online.” He reflected on the need to view situations 
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from a different lens: “...this student has maybe three other siblings that have to be online at the same time. What 
else? A little bit of humor is needed. It isn't that the student is not motivated, he is just always worried and has 
anxiety. They have other siblings who are online or parents who are working online at the same time and then 
having to log off and on so often.” 


Several faculty recounted challenges engaging students due to technology, cultural norms, and personality 
differences. Deficient internet access can result in loss of significant synchronous learning opportunities. “There 
are a lot of excuses. Well, my microphone is not working. This is not working. So | have to consistently say, hey are 
you online. So and so are you still here? So and so, sic let me know in the chat. Just something | think maybe a 
little bit of a change of mindset about online learning in regards to from a cultural perspective would be very 
helpful because it is distracting when you only have two or three out of thirty five students actually even saying 
anything in class or discussing anything or presenting anything there, you know.” And another faculty agreed, 
commenting “I have difficulty with getting the students to actually speak up.” 


Recommendation: Ensure availability and communication of learning and technology 
requirements as well as available assistance to support faculty and students in online 
learning environments. 


Target Faculty, Students, Advisors, Information Technology Staff, Instructional Designers 

Audience 

Needs e Increase faculty and students’ knowledge of technologies and online learning best 
practices 


e Increase training for faculty to increase engagement from all students 

e Instructional designers to assist faculty in developing courses and using tools/apps that 
increase engagement and limit distractions 

e Increase technology, including assistive technology access 

e Provide aseamless process for faculty and students to request accommodation or 
other support to create equitable learning opportunities 


Initiatives Ensure technology training, tools/hardware, and resources to support best practices and 
academic success are available to all faculty and students engaged in online learning 


Goals Enrich and support the student and faculty online learning experience to ensure successful 
student outcomes 
Actions e Extend the breadth and depth of orientation topics 


e Introduce Online Learning Readiness assessment prior to enrollment in online courses 

e Provide training opportunities using assistive technologies 

e Ensure instructional design support provides faculty the tools and best practices that 
support engagement from diverse student groups 

e Invest in technology and training to ensure all faculty and students have access to 
accommodations or assistive technologies to deliver an equitable learning 
environment including the course content and associated resources 


CSFs e Ensure early communication of the availability of instructional design support, assistive 
technology, other support services to students and faculty 
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KPIs e Students’ surveys and feedback report that the necessary tools, training, resources, 
and assistance was available for them to participate in the online learning environment 


Subdimension: Accessibility 


Measures of satisfaction with accessibility both pre- and post-COVID, among both administrators/staff and faculty 
reflect that the majority of participants are overall satisfied or very satisfied. It is of note that both satisfied and 
very satisfied responses decreased post-COVID (though satisfied responses decreased only slightly) while neutral, 
unsatisfied, and very unsatisfied responses increased (Figure 68). 
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As discussed in the previous section, 30% to 40% of faculty, students, and administrators/staff disagreed, or were 
uncertain, about the availability of tools and other resources to assist students challenged by disabilities. Yet, they 
reported that they believed the institution’s online courses demonstrated compliance with accessibility standards 
at high rates, with 80% administrators/staff, 82% of faculty, and 75% of students agreeing or strongly agreeing. 


The Ministry of Higher Education is a centralized authority responsible for directing university 

education in accordance with the adopted policy, supervising the development of university education in all 
sectors and formulating rules and regulations for compliance by all institutions of higher learning. Since higher 
education in Saudi Arabia has undergone tremendous growth over the past decade, it is vital to monitor laws and 
establish policies that govern disability access to online courses. 
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Recommendation: Ensure appropriate accessibility needs are in place through 
investigation of current practices. 


Target 
Audience 


Ministry of Education, Administrators, Staff, Information Technology Staff, Instructional 


Designers, Faculty 


Needs 


Investigate student and faculty needs 

Monitor legal requirements 

Explore assistive and adaptive technology 

Establish policies and practices to support course designers and provide 
accommodations for students and faculty with a disability 

Ensure accessibility features and functions are built in from the beginning, rather than 
approached retroactively 


Initiatives 


Ensure online learning practices adhere to accessibility laws, policies, and standards 


Goals 


Provide an equitable learning environment for students and faculty that follows best practices 


for digital learning 


Actions 


Form an Accessibility Task Force composed of IT staff, Faculty, college 
Administrators/Staff to investigate current technology and capacity for accessibility 
Provide recommendations to the appropriate university administrators and the 
Ministry of Education 

Make use of UDL 


CSFs 


Laws, policies, and practices are in place to ensure that the technology and resources 
for online courses is available to faculty and students 


KPIs 


The Accessibility Task Force surveys the current online learning environment and submits its 
findings and recommendations to the appropriate administrators, departments, and divisions 


of government 


Subdimension: Compliance Standards 


Measures of satisfaction with compliance standards both pre- and post-COVID, among both administrators/staff 
and faculty reflect that the majority of participants are overall satisfied or very satisfied. It is of note that both 
satisfied and very satisfied responses decreased post-COVID (though satisfied responses decreased only slightly) 
while neutral, unsatisfied, and very unsatisfied responses increased (Figure 69). 
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As referenced in the accessibility subdimension, all groups report agreement that the university’s online courses 
meet compliance with accessibility standards (Administrators/Staff: 80%, Faculty: 82%, Students: 75%). 


Additional data regarding institutional compliance was not available in the data. Future directions for compliance 


may address issues of economic competitiveness and online learning and institutional leaders should engage 


administrators, faculty, staff, students, and technology and content vendors as they prepare and implement their 


institutional compliance goals. 


Recommendation: Include all stakeholders including administrators/staff, faculty, 
Instructional Designers, and educational technology vendors in the implementation of 


compliance goals. 

Target Administrators, Staff, Information Technology Staff, Faculty, Educational Technology 
Audience Vendors 
Needs e Include all stakeholders in the implementation of compliance standards 

e Provide training about compliance standards and goals 

e Communicate compliance goals to the university community 
Initiatives Verify that all university personnel and staff are aware of and maintaining compliance goals 
Goals Provide a secure, trustworthy learning environment for the university community 
Actions e Establish annual review of the institution’s compliance goals 

e Implement compliance training for administrators/staff, faculty, and students 
CSFs Implementation of training and annual review of compliance goals 
KPIs e Successful completion of compliance training 


e Effective communication of compliance goals 
e Completion of annual review of compliance standards and goals 


92 i ________ 


Subdimension: Innovation 


Measures of satisfaction with student support innovation both pre- and post-COVID, among both 
administrators/staff and faculty reflect that the majority of participants are overall satisfied or very satisfied. It is 
of note that both satisfied and very satisfied responses decreased post-COVID while neutral, unsatisfied, and very 
unsatisfied responses increased (Figure 70). 
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The data indicates that many public institutions had provided online learning courses prior to the COVID-19 rapid 
transition to online course delivery; however, it seems that the majority of private institutions had yet to build 
online learning capacity at their institutions. Three-quarters of administrators/staff (75%) and faculty (75%) 
agreed that their institutions are supporting students in innovative ways, such as relevant communication modes 
or through Al communication tools. Students reported slightly less support (62%), with almost two-thirds 
agreeing that their campuses made use of innovative tools. 


Astaff member from a private university explained that the campus use of an online ticketing system that 
provided students easy access to one space to request any type of assistance: “So most of the services are 
provided online except, as | said, for student activities.” One private university faculty shared their experience with 
the engagement vs disengagement challenge of incorporating video, communication, and social media 
applications in online courses. He commented: “During the lecture they text - that is the direct interference. We 
allow them to open the apps if they have to answer... But even during the delivery of lectures they continue to 
communicate with you. The class used the WhatsApp group but students would use apps during class and were 
not engaged. It was a distraction.” 
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Recommendation: Invest in educational technology tools and research to stimulate 
innovative online educational practices and support services. 


Target Administrators, Staff, Faculty, Students, Information Technology Staff, Instructional Designers, 
Audience Educational Technology Vendors 
Needs e Create capacity and scale for online learning 


e Make use of third-party vendors as appropriate and cost effective 

e Improve the online learning experience for students 

e Investigate the delivery of health and well-being check-ins, automated feedback, and 
technology training through vendors 

e Make use of shared tools through strategic approach to learning tools adoption 


Initiatives Develop policies and practices to investigate innovative tools and implement an adoption 
strategy for tools and practices 
Goals Provide a secure, engaging learning experience that leads to positive student learning 
outcomes 
Actions e Create a Learning Innovation Taskforce (or new unit) to investigate tools and 
pedagogy 


e Develop policies and practices that deliver cost-effective, innovative tools that 
provide capacity and scale for online learning 


CSFs Increased capacity for online learning and delivery of student support resources 


KPIs Students’ surveys and feedback report positive interactions with course tools and provide 
opportunities for timely, engaging faculty and peer interactions 


Dimension VII: Training & Support 


The Training and Support dimension measures five subdimensions including Technical Assistance, Professional 
Development, Orientation, Mentoring, and Innovation. 


Prior to COVID, satisfaction measures in the training and support dimension indicate overall congruence among 
administrators, staff, and faculty. The study revealed two primary areas that would have benefitted from 
additional readiness prior to the pandemic: technical assistance (particularly for students) and professional 
development. These items are detailed below, and results indicate that priority was placed on these readiness 
items quickly, as evidenced by post-COVID satisfaction measures. 


Overall findings indicate that the majority of administrators, staff, and faculty felt that those teaching online 
courses had sufficient training and support for the move to remote learning. These activities often include 
workshops provided by their institutions. There are some small discrepancies between perceptions by 
administrators/staff and faculty to the degree of their agreement, but they overall trend similarly. While students 
were not asked to rate items from these subdimensions since it would be unlikely that they would have insight 
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into these areas, interview questions illuminate concerns that faculty were not as adequately prepared nor had 
sufficient technical assistance to support students and have high quality course delivery. 


Subdimension: Technical Assistance 


Measures of satisfaction with technical assistance both pre- and post-COVID, among both administrators/staff and 
faculty remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied. 
It is of note that post-COVID, satisfied responses decreased slightly while very satisfied responses increased 
(Figure 71). 
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The overwhelming majority of administrators, staff, and faculty felt that those teaching received sufficient 
technical assistance for course development and online teaching. Administrators and staff reported that they 
agreed or strongly agreed at a rate of 81.4% where faculty similarly reported at a rate of 81.9% (Figure 72). 


Technical assistance is provided for faculty during online 
course development and online teaching. 
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While students did not respond to this subdimension in the survey, public and private school students shared that 
faculty ran often into technical issues, such as around recorded lectures, and that overall everyone needed more 
technical assistance to use new technologies. Some had great responses emailing technical support offices, but 
one shared that “You just have to email the Dr. and tell them it’s crashed and deal” and if they do not accept the 
excuse, “there’s nothing else that you can do.” Another mentioned that they “did not have any technical 
assistance.” Multiple students said that there was often a substantial time gap in responses to technical issues, 
with one sharing that they felt it is because there were “a lot of questions from students” at that time. 


Recommendation: Administrators, staff, and faculty generally felt that there was 
sufficient technical assistance, but students sampled for this study tended to have a more 
negative response to technical issues and support. 


Target Administrators, Staff, Faculty, Students 
Audience 
Needs Improve communication and technical assistance mechanisms for students 
Initiatives Create a centralized support space through a website; Resources explicitly shared and stressed 
by faculty in syllabi and online course, and the institution 
Goals Remove barriers for course progress and develop resources for common issues 
Actions e Create an advisory team of stakeholders across campus including student members 
to guide development of needed resources 
e Create a centralized website with curated resources (e.g., guides, videos, frequently 
asked questions) and support mechanisms (ex: live chat, chatbot) 
e Communicate resources through institutional channels, online courses, and syllabi 
CSFs e Stakeholder expertise and listening to student needs 
e Centralized website available with curated resources and support mechanisms, and 
updated to ensure current 
e Resources shared through institutional channels, online courses, and syllabi 
KPIs e Website piloted and shared for feedback 


Future student technical issues curated to update website 
Each syllabus and online course has common support language 
Institution shares resources through a variety of channels 
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Expert Recommendation: Provide adequate technical assistance for faculty. 


Target Administrators, staff, faculty 
Audience 
Needs Improve communication and technical assistance mechanisms for faculty 
Initiatives Create a centralized support space through a website; Resources explicitly shared by staff 
Goals Remove barriers for faculty teaching in an online modality 
Actions e Create an advisory team of stakeholders across campus including faculty to guide 
development of needed resources 
e Create a centralized website with curated resources (e.g., guides, videos, frequently 
asked questions) and support mechanisms (ex: live chat, chatbot) 
e Communicate resources through institutional channels 
CSFs Centralized website available with curated resources and support mechanisms, and updated 
to ensure current 
KPIs e Website created 


e Faculty technical issues curated to update website 


Subdimension: Professional Development 


Measures of satisfaction with professional development both pre- and post-COVID, among both 
administrators/staff and faculty remained stable, with the majority of participants indicating that they are overall 


satisfied or very satisfied (Figure 73). 
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Overall, the majority of administrators, staff, and faculty felt that those teaching received training assistance, and 


support to prepare for course development and online teaching. Administrators and staff reported that they 
agreed or strongly agreed at a rate of 81.4% where faculty reported at a higher rate of 83.9% (Figure 74). Similarly, 
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faculty reported a higher rate of 81.2% that they received ongoing professional development for online teaching 
and learning over administrators/staff who reported at 78.4% (Figure 75). 


The institution ensures faculty receive training, 
assistance, and support to prepare for course 
development and teaching online. 
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Faculty are provided ongoing professional development 
related to online teaching and learning. 
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Similarly, the majority also felt that faculty had sufficient time for professional development; however, there was a 
slight difference between administrators/staff and faculty. Administrators/staff reported at 73.7% while faculty 
reported at 69.8% (Figure 76). Additionally, while most administrators, staff and faculty felt that clear standards 
were established for faculty engagement and expectations concerning online teaching, administrators/staff 
reported at 78.4% while faculty reported lower at 73.6% (Figure 77). These findings illuminate a small discrepancy 
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between administrators/staff and faculty as it related to faculty development. They also are the lowest rated of 
training and support subdimensions. 


The institution provides adequate time for faculty to 
engage in professional development. 
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Figure 76 


Clear standards are established for faculty engagement 
and expectations concerning online teaching. 
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An administrator at a public institution mentioned that they had concerns about faculty having “advanced skills 
on how to teach properly” online nor “new tech tricks to make their classes lovely,” instead focusing solely on 
PowerPoint presentation and conducting class “the same way that they are used to...teaching in front of their 


students inside their classrooms.” A faculty member at a private institution similarly stated that “some of our staff 
are not tech savvy. | think they would need more workshops. Also...we need to expand our horizon when it comes 


to using this technology.” 
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Anumber of administrators at public institutions also shared the desire for faculty certification for training with 
one sharing that teachers at one of the international universities “will be given a certificate” for completion. A 
faculty member at a private institution also shared that their institution “conducted several workshops to the 
faculty and different sessions of different levels to accommodate the majority of the faculty. Some of them don’t 
have experience with any virtual classes. So they have them as a beginner session and they had an intermediate 
session and advanced sessions...It was very good. Very challenging because in one week we’ve done everything 
and the next week we just started our online classes.” 


While students did not respond to this subdimension in the survey, students at public institutions felt that they 
“need a faculty that is well-trained and updated with technology... for the long run... after the coming semester.” 
One student at a private institution shared that their faculty did not know how to use technology and “some of 
them were even asking students what to do.” They continued that the transition to online “was a mess” and “a lot 
of them weren’t prepared” to move online. 


Recommendation: Overall, administrators, staff, and faculty tend to agree that they had 
sufficient training and support. Given that there were some discrepancies between 
administrators/staff and faculty on a number of different areas, increased communication 
between those stakeholder groups could take place to improve rating by faculty. In 
addition to faculty, students had a far more negative perspective in their interview 
responses and can help close the feedback loop. 


Target Administrators, Staff, Faculty 

Audience 

Needs Improved communication between stakeholder groups 

Initiatives Create mechanisms for a feedback loop 

Goals Improve communication and better understand where pain points exist and how to better 


address faculty and student needs to improve overall course outcomes 


Actions e Create opportunities for stakeholder feedback (e.g., post-event surveys, focus groups, 
town hall meetings) 

e Conduct a needs assessment to help illuminate areas of where faculty need more 
support, resources, etc. 

e Create a mechanism for students to share their perspectives (e.g., town hall meetings, 
focus groups, survey) 


CSFs e Broad faculty participation in needs assessment 

e Timely creation of feedback loops that are maintained and communicated each 
semester 

e Feedback implemented 

KPIs e Response rate of 50% or greater 


e Improved faculty satisfaction with professional development and student satisfaction 
with courses 
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Recommendation: Time proved to be a challenge for those needing professional 
development during the move online; however, this challenge is not unique to the time 
affected by COVID (although it further exacerbated the challenges). A recommendation 
might be moving a long synchronous training into a modular, asynchronous format. Doing 
so provides opportunities for faculty to complete the work when they have time and in 


smaller chunks. 
Target Administrators, Staff 
Audience 
Needs Increase opportunity for faculty to complete professional development and remove barriers for 
support resources 
Initiatives Transition synchronous training to be more asynchronous and curate resources in a centralized 
space 
Goals Provide greater flexibility and more options to complete professional development and remove 
barriers for support resources 
Actions e Break longer synchronous training sessions into more manageable parts that can be 
completed over a longer period of time 
e Create short courses using the resident learning management system (LMS) (ex: 
Blackboard) around specific training topics to create a space for engagement, practice 
with tools and features, and share ideas, ask questions, and engage with the content 
e Curate, post, and communicate resources pertaining to moving teaching online in one 
space (such as a website for a Center for Teaching and Learning) 
CSFs e Opportunities for faculty to ask questions in a synchronous format (e.g., weekly live 
session or virtual office hours) 
e Modular content that can be completed in shorter chunks of time 
e Technical support structure for short courses 
e Advisory group of faculty and staff to provide feedback on curated resources 
e Faculty are made aware of resources through a variety of channels (e.g., departments, 
emails, etc.) 
KPIs e Increased faculty participation in professional development 


Improved student outcomes (depending on topic) 


Expert Recommendation: Provide regular opportunities for professional development 
specific to online learning. 


Target Audience 


Faculty 


Needs Regular access to professional development 
Initiatives Faculty training on course design and teaching online 
Goals Provide regular access to professional development specific to online learning 
Actions e Identify professional development needs 

e Provide workshops on specific needs (either in-house or through provider) 
CSFs Faculty better trained to teach online 
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KPIs Faculty provided with access to quality training 


Subdimension: Orientation 


Measures of satisfaction with orientation both pre- and post-COVID, among both administrators/staff and faculty 
remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied (Figure 
78). 
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Figure 78 


The overwhelming majority of administrators, staff, and faculty felt that those teaching received a sufficient 
technology and instructional design orientation. Overall, administrators/staff reported at 79.6% and faculty 
reported at 78.2%; however, faculty reported 4.3% fewer on strongly agree than administrators and staff with 
higher agree (Figure 79). 
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One faculty member at a private institution shared that they created a e-learning committee and conducted 


“orientation for the faculties and students for how to handle the online courses and how to teach online.” An 


administrator also shared that their institution had a mandatory “Blackboard orientation” for students. 


Recommendation: Most faculty shared that they received an orientation and it is not fully 
clear why those who reported they disagreed did not, or why a number were neutral. 
Throughout the Kingdom of Saudi Arabia, many technologies will be the same across 
institutions. The Ministry or NELC could work with some institutions to identify these 
technologies and develop and share common orientations that could be used by anyone 


within the Kingdom. 
Target Administrators, Staff, Ministry of Education, National E-Learning Center 
Audience 
Needs Improve access to orientations for widespread/common technologies 
Initiatives Develop and share common orientation modules for institutions 
Goals Decrease resistance and challenges among faculty for online learning technologies and 
improve student outcomes 
Actions e Identify widespread technologies and determine most critical to develop 
e Partner with different institutions to develop and pilot orientation modules 
e Publicly host and share modules with institutions 
CSFs e Most critical technologies identified 
e Periodically updated to include new technologies 
e Piloted with institutions to provide feedback 
e Orientations easily accessible and shared with key personnel at institutions 
KPIs e Number of modules developed and updated periodically 


Number of modules accessed and implemented by institutions 
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Subdimension: Mentoring 


Measures of satisfaction with mentoring both pre- and post-COVID, among both administrators/staff and faculty 
remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied. It is of 
note that post-COVID, satisfied responses decreased slightly and neutral responses increased (Figure 80). 
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Overall, most administrators, staff, and faculty felt that those teaching received sufficient peer mentoring 
resources. Administrators/staff reported at 77.8% with faculty reporting at 74%. Similarly, faculty reported more 
neutral indications and fewer strongly agreed (Figure 81). 


There are peer mentoring resources available to faculty 
members teaching online courses. 
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Recommendation: While faculty generally indicated that they received sufficient peer 
mentoring resources, almost a quarter did not. Peer mentoring provides opportunities to 
support each other through the shift to remote and fully-online teaching and learning as 
well as basic life needs and continuing to grow as an academic. A virtual mentoring 
program coordinated centrally could provide opportunities for greater connection and 


support. 
Target Administrators, Staff, Faculty 
Audience 
Needs Preserve and improve opportunities for peer mentoring 
Initiatives Develop a virtual mentoring program 
Goals Provide opportunities for faculty to continue to receive support and guidance during remote 
teaching and learning, including maintaining academic momentum 
Actions e Communicate with faculty to identify mentoring needs 
e Pair mentors with mentees based on needs 
e Provide resources and incentives where possible and appropriate 
CSFs e Needs identified 
e Faculty volunteers to be mentors and mentees 
e Means for incentivizing participation, if possible 
e Feedback loop to improve program where possible 
KPIs e Number of mentors and mentees 


Academic output (e.g., publications, grants, etc.) 
Faculty retention 
Faculty health and wellbeing 


Subdimension: Innovation 


Measures of satisfaction with training and support innovation both pre- and post-COVID, among both 
administrators/staff and faculty remained stable, with the majority of participants indicating that they are overall 
satisfied or very satisfied. It is of note that post-COVID, satisfied and very satisfied responses decreased slightly 
and neutral responses increased (Figure 82). 
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Innovation 
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The majority of administrators, faculty, and staff felt that those teaching were informed about emerging 
technology and new tools. Administrators/staff reported at 79.6% and faculty similarly reported at 79.9% (Figure 
83). 


Faculty are informed about emerging technologies and 
the selection and use of new tools. 
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An administrator at a private institution shared that they have a “Center of Excellence in Teaching and Learning 
that offers training for faculty...new ways of teaching methods, innovation, and teaching assessment planning.” 
They continued that faculty could “submit a portfolio describing their ways of teaching and which way they 
became innovative” and encouraged them to “be innovative on their own.” 
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Recommendation: A fantastic way to support the awareness and use of new technologies 
is through a faculty learning community program. These communities can take place 
organically, led by a university staff member or faculty member, or can be coordinated 
centrally through divisions of academic/faculty affairs or teaching and learning centers. 


Target Administrators, Staff, Faculty 

Audience 

Needs Greater awareness and use of innovative technologies 

Initiatives Develop a faculty learning community program 

Goals Build capacity around online learning tools and approaches 

Actions e Bottom-up: faculty/staff self-organize into a community (or multiple communities) 


e Top-down: institution recruits faculty/staff into communities and provides centralized 
way to coordinate the groups 


CSFs e Bottom-up: Groups set expectations, regular meetings, participants share resources 
and ideas 

e Top-down: Clear expectations for program, regular meetings, a point person to 
coordinate the group and ensure all needs and outcomes met, incentives for 
participation (stipends, resources, awards/recognition, etc.), recruitment from all 
departments to grow champions 


KPIs e Number of groups 
e Participation in groups 


e Diffusion of knowledge and skills 


Dimension VIII: Evaluation & 
Continuous Improvement 


The Evaluation and Continuous Improvement dimension measures five subdimensions including Evaluation of 
Course Outcomes and Program Quality, Student Satisfaction, Faculty Satisfaction, Staff Satisfaction, and 
Innovation. 


Prior to COVID, satisfaction measures in the evaluation and continuous improvement dimension indicated 
positive responses, through survey and interview data indicate that readiness for a fully online environment 
needed improvement across all subdimensions, with the exception of student satisfaction. This is evident in that 
satisfaction decreased while dissatisfaction and neutral perceptions increased post-COVID, and is supported by 
survey and interview data, detailed below. 


Overall findings indicate that the majority of evaluation and continuous improvement activities are focused on 
course outcomes and program quality. Both survey and interview data suggest that these initiatives are 
comprehensive and multi-layered, encompassing multiple evaluation measures and types across course, 
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program, and institutional levels, as well as including a variety of stakeholders for both conducting evaluations 
and planning based on results. Findings indicate a discrepancy between administrators/staff and faculty with 
regard to the use of innovative analytics for programmatic decision-making, and faculty responses also indicated 
potential improvement opportunities for peer evaluation at the course level. Additionally, student responses 
indicate that there are potential improvement opportunities for providing feedback on both course improvements 
as well as the effectiveness of instruction and quality of online learning materials. 


Subdimension: Evaluation of Course Outcomes and 
Program Quality 


Measures of satisfaction with evaluation of course outcomes and program quality both pre- and post-COVID, 
among both administrators/staff and faculty reflect that the majority of participants are overall satisfied or very 
satisfied. It is of note that both satisfied and very satisfied responses decreased post-COVID while neutral, 
unsatisfied, and very unsatisfied responses increased (Figure 84). 
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Administrators and staff reported at a rate of 73.8% that they believe there are procedures and policies for 
evaluating and improving programs, and both administrators/staff and faculty agreed at rates of 75% or above 
that there are established processes, standards, and course-level alignment and evaluation in place. Additionally, 
faculty responded at similarly high rates that overall course-level evaluations, as well as evaluations of syllabi, 
teaching materials, course assignments and activities, and teaching performance are implemented. Interviews 
confirm these findings with statements such as: “We used to have and still have, of course that assessment 
committee that reviews all the courses and makes sure to align all the outcomes of the courses with the outcomes 
of the program. In addition, where this is to maintain the quality of the courses and for the program at all. And also 
we have the students’ feedback at the end of each course, at the end of each semester, and of course we have peer 
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review for the quizzes,” and another that stated “Accredited assessment through different levels. Course level, 
program level, institution level. Every course is assessed through the student point of view from their own course 
surveys. At the end of the summer, but also regularly during the semester, there are class visits from other peer 
reviews, from the leadership in the department. And for the program, all these are incorporated into assessment 
of the end of the semester, into assessment of learning outcomes, how much students learned in the course. Also 
across the programs we study and we quantify how much learning happened in a program from all the students. 
And then we can conclude from the institution level.” 


As noted above, there are discrepancies between administrators/staff in the implementation and use of peer 
evaluation, with faculty responding at a rate of 65.6% that agreed or strongly agreed that these methods are used, 
and 12.7% indicating that they disagreed or strongly disagreed (Figure 85). There is also a slight gap between 
faculty and administrators regarding the use of innovative analytics for programmatic decision-making, with 
74.7% of administrators/staff agreeing or strongly agreeing, and 67.9% of faculty agreeing or strongly agreeing 
(Figure 86). 


Faculty are expected to participate in peer evaluation of 
their online courses as well as the courses of others. 
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The institution uses innovative developments in analytics 
to guide course and programmatic decisions about online 
teaching and learning. 
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Recommendations in this area are to continue following established processes, procedures, and policies with 
regard to course and program evaluation, and to regularly review and update them in alignment with quality 
standards and best practices. It is essential to continue including a variety of evaluation types as well as a diverse 
representation of stakeholders in evaluation planning and processes. Regarding innovative analytics for 
programmatic decisions, it is possible that increased transparency and communication about these initiatives 
may benefit the opinions, beliefs, and perceptions of faculty, and including faculty in these initiatives could also 
make a difference. The other discrepancy in this section, faculty peer evaluation, could benefit from clear 
practices and expectations on peer evaluations, as well as explicit processes and procedures related to peer 
evaluation, and appropriate professional development on the development, implementation, and use of such 
practices. 


Recommendation: Ensure adherence to established processes, procedures, and policies 
on course and program evaluation, including regular review and updating. 


Target Faculty 
Audience 
Needs Develop and implement clear processes, procedures, and policies on course and program 


evaluation, including regular review and updating 


Initiatives Processes, procedures, and policies on course and program evaluation, including review and 
updating, are developed and implemented 


Goals Maintain and/or increase a high level of adherence to processes, procedures, and policies for 
course and program evaluation, and to continually improve these items through regular review 


and updating 
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Actions 


Develop and implement course and program evaluation processes, procedures and 
policies 

Regularly review and update course and program evaluation processes, procedures, 
and policies 


CSFs 


Course and program evaluation processes, procedures, and policies are in place 
Course and program evaluation processes, procedures, and policies are regularly 
reviewed and updated as appropriate 


KPIs 


Courses and programs are evaluated in alignment with established processes, 
procedures, and policies 


Course and program processes, procedures, and policies are reviewed and updated as 


appropriate on a regular schedule 
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Recommendation: Provide communication and transparency to faculty, and involve 
faculty when possible in initiatives regarding analytics for programmatic decisions. 


Target Faculty 
Audience 
Needs Increased communication, transparency, and involvement of faculty with regard to analytics 


used for programmatic decision-making 


Initiatives 


e Communications to faculty regarding the analytics included in programmatic 
decision-making 

e Inclusion of faculty in programmatic analytics and decision-making when possible 
and appropriate 


Goals Increase faculty awareness, participation, and buy-in of programmatic decision-making 
Actions e Communicate with faculty regarding analytics used for programmatic decision- 
making 
e Include faculty, when possible, in programmatic decision-making 
CSFs e Communications are provided to faculty regarding programmatic decision-making 
e Faculty are included in programmatic decision-making 
KPIs e Faculty indicate awareness of analytics used for programmatic decision-making 


e Faculty participate in programmatic decision-making processes 


Recommendation: Develop and implement practices and expectations regarding faculty 
peer evaluation, including explicit processes and procedures as well as appropriate 
professional development on the development, implementation, and use of such 


practices. 
Target Faculty 
Audience 
Needs Develop and implement practices and expectations regarding faculty peer evaluation, including 
processes and procedures as well as professional development on development, 
implementation, and use 
Initiatives Implement faculty peer evaluation, and provide related professional development 
Goals Increase the use of effective peer evaluation for faculty in order to support continued 
evaluation and development of effective teaching and learning practices 
Actions e Develop and implement practices and expectations regarding faculty peer evaluation 
e Provide professional development on peer evaluation to faculty 
CSFs e Clear practices and expectations for faculty peer evaluation are in place 
e Professional development opportunities on peer evaluation are available to faculty 
KPIs e Faculty peer evaluation is effectively employed 


e Faculty participate in professional development opportunities related to peer 
evaluation 
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Subdimensions: Student Satisfaction, Faculty 
Satisfaction, and Staff Satisfaction 


Measures of satisfaction with student satisfaction both pre- and post-COVID, among both administrators/staff and 
faculty remained stable, with the majority of participants indicating that they are overall satisfied or very satisfied. 
It is of note that post-COVID, satisfied responses decreased slightly while very satisfied responses increased 
(Figure 87). 


Student Satisfaction 
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Measures of satisfaction with faculty satisfaction both pre- and post-COVID, among both administrators/staff and 
faculty indicate that the majority of participants are overall satisfied or very satisfied. It is of note that post-COVID, 
satisfied and very satisfied responses decreased while very neutral, unsatisfied, and very unsatisfied responses 
increased (Figure 88). 


Faculty Satisfaction 
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Measures of satisfaction with faculty satisfaction both pre- and post-COVID, among both administrators/staff and 
faculty indicate that the majority of participants are overall satisfied or very satisfied. It is of note that post-COVID, 
satisfied and very satisfied responses decreased (though satisfied responses decreased only slightly) while very 
neutral, unsatisfied, and very unsatisfied responses increased (Figure 89). 


Staff Satisfaction 
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Administrators/staff and faculty were queried as to whether stakeholder assessments for students, faculty, and 
staff were in place. The majority, 75.3%, agreed or strongly agreed with this statement, with less than 8% 
disagreeing or strongly disagreeing (Figure 90). 


Student responses indicated that opportunities exist for feedback on both course improvements (62.1% agreed or 
strongly agreed that they are present; 20.6% disagreed or strongly disagreed), and evaluations on the quality of 
instruction and online materials (66.9% agreed or strongly agreed that they have these opportunities; 15.6% 
disagreed or strongly disagreed). Interestingly, interview data repeatedly indicates that student evaluations and 
feedback are collected. 
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Recommendations include communication of evaluation opportunities (faculty, students, and staff) and their 
purpose, as well as communications about how such feedback is incorporated into course and program 
improvements. Intentional development of evaluation and feedback opportunities to directly impact ongoing 
improvements is also recommended. 


Recommendation: Communications are provided to faculty, students, and staff regarding 
evaluation opportunities and their purpose, including information on the use of such feed 
for course and program improvements. 


Target Faculty, Staff, and Students 

Audience 

Needs Communications to faculty, students, and staff regarding evaluation opportunities, their 
purpose, and how feedback is incorporated into course and program improvements 

Initiatives Create a communication plan for faculty, staff, and students regarding evaluation 
opportunities, purposes, and uses 

Goals Increase awareness and opportunities for faculty, staff, and student evaluations 

Actions Develop and implement a communication plan regarding evaluation opportunities, purposes, 
and uses for faculty, staff, and students 

CSFs Communications are provided to faculty, staff, and students regarding evaluation 
opportunities, purposes, and uses 

KPIs Faculty, staff, and students participate in evaluation opportunities 
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Subdimension: Innovation 


Measures of satisfaction with faculty satisfaction both pre- and post-COVID, among both administrators/staff and 
faculty indicate that the majority of participants are overall satisfied or very satisfied. It is of note that post-COVID, 
satisfied and very satisfied responses decreased (though very satisfied responses decreased only slightly) while 
very neutral, unsatisfied, and very unsatisfied responses increased (Figure 91). 
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Innovation in the evaluation and continuous improvement dimension includes the continual implementation of 
appropriate assessment methods for the evolving online higher education landscape in the KSA, as well as explicit 
attention to the development, implementation, and communication regarding innovative analysis methods used 
to inform improvements (as noted in a previous section). 


Expert Recommendation: Incorporate best practices. 


Target Administrators, staff, faculty 

Audience 

Needs Access to research and international experts 

Initiatives Implement best practices identified by online learning experts 

Goals Ensure quality of online learning by benchmarking against best practices 

Actions e Review quality standards (rubrics, scorecards, research, etc.) shared by online 


learning experts 
e Apply best practices to online learning courses and programs 


CSFs Improved quality of online learning 


KPIs e Use of rubrics or scorecards to evaluate quality of online learning 


e Adoption of best practices 
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Pressing Challenges and 
Opportunities 


In Part Ill of the Higher Education survey, respondents were asked to choose from a list of 28 items that they 
perceived or experienced as challenges for online learning at their institutions and could check all that they felt 
applied to their context. The top five reported challenges include Student Online Readiness (61.9%), the Quality of 
Online Courses (50.2%), Access to the Internet (44.4%), Familiarity with Online Learning (43.2%), and Technological 
Infrastructure (38.1%). Some other relevant findings include Student Engagement with their Course (37.3%), Time 
Management (34.6%), Availability of Professional Development Funding (34.5%), Faculty-Student Engagement 
(34.4%), Extra-Curricular Group and Campus Engagement (33.4%), and the Flexibility of Student Academic Support 
Services (31.5%). The items that had the fewest noted challenges include Shared Governance Policies/Processes 
(19.1%), Spaces for Learning (18.4%), Area-Specific Leadership (16.5%), and the Cost of Course Materials (15.2%). 
Additional findings are all located in Appendix IX: Pressing Challenges. Overall, respondents considered leadership 
factors as less of a challenge, where technology, online courses and readiness, and engagement were more of a 
challenge. 


In addition to challenges, respondents were asked to rank 19 potential opportunities for future investment of 
resources to support online learning. The following findings are ranked from highest to lowest. 


Faculty/Instructor Training (M = 4.37, SD = 3.790) 
Leadership (M = 4.41, SD = 4.698) 
Institutional Policies (M = 4.72, SD = 4.122) 
Student Academic Support Services (M = 5.49, SD = 3.501) 
Student Online Readiness (M=5.50, SD = 3.409) 
Technology Infrastructure (M = 6.52, SD = 3.382) 
Instructional Design (M = 6.58, SD = 3.088) 
Student Technology Resources (M = 7.98, SD = 2.961) 
Faculty Technology Resources (M = 8.14, SD = 3.110) 
. Online Work Skills (e.g., Microsoft Word) (M = 9.93, SD = 3.092) 
. Mental Health Services (M = 11.43, SD = 3.159) 
. Open Educational Resources (M = 12.49, SD = 2.866) 
. Physical Health Services (M = 12.50, SD = 3.026) 
. Communication (M = 12.94, SD = 3.288) 
. Innovative Thought Leadership (M = 13.68, SD = 3.683) 
. Student Engagement with Course (M = 14.24, SD = 3.879) 
. Accessibility and Universal Design Resources (M = 15.46, SD = 3.564) 
. Governance, Policies, and Processes (M = 15.65, SD= 4.774) 
. Opportunities for Peer Learning (M = 16.81, SD = 4.269) 
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While there are some similarities between challenges and opportunities, there are also some notable differences. 
Where some leadership factors were considered less of a challenge, respondents felt that more investment is 
needed. Student engagement was seen as a significant challenge, but was rated low for additional investment. 
Student online readiness, training for faculty for building online courses, and technology infrastructure all rate as 
higher on both. 


Summary of Recommendations 


The recommendations resulting from the quantitative and qualitative analyses of this study indicated a number of 
needs and actions that will allow the Kingdom of Saudi Arabia to effectively increase and continuously evaluate 
and improve the quality, effectiveness, and capacity of online higher education. Several expert recommendations 
in addition to the key actions are also provided to further support online education initiatives and Vision 2030. 
Overall, findings indicate a high level of consensus among administrators/staff and faculty across all eight 
dimensions. Areas in which there are differences among administrators/staff and faculty are clearly indicated, 
along with potential reasons for their presence and recommendations to address them. While a number of items 
indicate that students are in alignment with administrators and faculty/staff, a clear gap is often present between 
students and administrators/staff. Recommendations include potential explanations and solutions to address 
these gaps. 


Recommendations are provided in detail in the narrative above and summarized in Appendix IV: Future Action 
Framework (including priority recommendations for COVID). 


An action plan for implementing recommendations within the context of readiness, challenges, responses, and 
impact is summarized in Appendix Il: Research Findings. A trend identified in this appendix is that there are many 
areas in which ad-hoc solutions to address challenges either illuminated or caused by the pandemic were put in 
place locally at institutional levels or independently by faculty and instructors. In order to most effectively 
implement the action plan, evaluation of the gaps addressed in this manner should take place, and determination 
of need for scaled processes and solutions as well as which should remain at independent levels should be 
completed. These issues are addressed through the recommended actions, but clear awareness of this trend is 
warranted. Additionally, many areas in this study reflected either increases or decreases in satisfaction post- 
COVID, though explicit actions that may have resulted in these changes are not indicated. It may be of value to 
further investigate the impetus for change in some of these areas based on selected priorities. Both during and 
following the pandemic, adjustments will need to be made to adapt to the new environment, meet immediate 
needs, and remain on track to accomplish the goals of Vision 2030. The OLC Evaluate tool can also help specific 
higher education institutions determine their campus-specific needs based on their unique contexts. 


As noted above, though gaps exist and are addressed in this study, the overall high and consistent rate of 
similarity in reported opinions, beliefs, and experiences reported not only provides indications for areas of 
improvement, existing areas of excellence are evident, and the findings demonstrate that there is a solid 
foundation upon which to build as the Kingdom of Saudi Arabia pursues its online education goals. 
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Considerations of 
Future Study 


Resulting from the development process of the survey and interview protocols and 
evaluation frameworks, as well as from the actual study findings, several 
considerations for expanded and additional future studies are recommended. 


This developmental study is comprehensive in scope and the establishment of an online learning evaluation 
framework, evaluation of pre-, peri-, and post-COVID contexts, practices, and needs and included both 
quantitative descriptive data and nuanced qualitative data. An expansion of this study into multivariate analysis 
could lend further exploration and indications for each dimension and subdimension, resulting in further 
recommendations for the development and implementation of effective practices, meeting stakeholder needs, 
and supporting proactive development of adequate capacity for current and future infrastructure, design, and 
delivery of online courses. Additionally, expanding on this work to include multivariate analyses may be valuable 
for the evaluation and continuous improvement of the evaluation framework. 


Developing and implementing longitudinal measures to follow this study is recommended for optimal evaluation 
of existing and evolving needs, and to provide clear markers of changes, successes, challenges, and opportunities. 


Additionally, consideration should be given to the development of several smaller studies designed to provide 
deep analysis of institution and stakeholder responses at more granular levels such as private institutions, public 
institutions, specialized institutions, and by further disaggregated stakeholder characteristics. These analyses 
would primarily benefit institutional-level stakeholders by providing recommendations specific to their individual 
environments and contexts. 


During both the development and analysis processes for the current study, a potential need for the design and 
completion of a similar study, focused on employment projections and employer needs, community needs, and 
partnership opportunities and needs emerged. The completion of such a study would benefit higher education 
stakeholders in the KSA, including administrators/staff, faculty, and students. The incorporation of such findings 
has the potential to proactively inform future course and program curriculum development as well as to assess 
and improve existing curriculum to best equip students for future employment success. Understanding 
community needs could potentially inform not just curriculum, but provide insight to administrators/staff and 
faculty regarding the student population and their holistic support needs (academic, financial, health, and social- 
emotional). Similarly, understanding the present and potential partnership opportunities could allow for the 
development and/or modification of strategic plans that include optimally effective and efficient structures 
spanning a wide variety of institutional, stakeholder, and Kingdom-level needs. 
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There are a number of expansions and/or additional studies that may be beneficial to higher education in the KSA. 
The primary recommendations described above were selected as those most likely to provide maximum benefit 
based on the development and analysis of the present study. 
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Appendices 


Appendix I: NELC Evaluation Framework 


Governance, Strategies, Policies, Processes, Resource Allocations, Periodic Review & Updating, Innovation 


2 | Curriculum 
Design & Planning 
a Instructional Design Methods & UDL, Alignment with Standards, Course Syllabi, Course Materials & Content, Innovation 


Online Teaching & 
Learning 


Communication, Engagement, Expectation Setting, Outcomes, Course Interaction, Feedback, Resources, Innovation 


Assessment Strategies, Assessment Processes, Assessment Methodology, Assessment Types, Innovation 


Operability, Centralized Online Education Infrastructure (SSO, LMS, etc.), Modality, Security, ITSM Compliance, Internet 
Access, Reliability, Coverage, Innovation 


Student Orientation & Support, Equity, Accessibility, Compliance Standards, Innovation 
Training & 

Support 

Technical Assistance, Professional Development, Orientation, Mentoring, Innovation 


Evaluation & 
Continuous 
Improvement 


Student Satisfaction, Faculty Satisfaction, Staff Satisfaction, Evaluation of Course Outcomes and Program Quality, 
Innovation 
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Appendix II: Research Methodology Framework Mapping 


SI.No 


il 


2 


Dimensions 


Leadership 


Curriculum 
Design & 
Planning 


Measurements 


Objectives 


Methodology 


Instruments 
(Data Source) 


Governance, Strategies, Policies, Process, Resource Allocation, Periodic Review and Updating, Innovation 


Objectives 


Pre COVID-19 
Readiness 


Baseline measures of 
subdimensions; what existing 


governance, strategies, policies, 


processes, resource availability 
and allocation, and innovations 
existed prior to COVID-19 for 
leadership? 

Qualitative; survey analysis and 
policy analysis 


Data collected via survey; 


policies provided for analysis by 


NELC 


Baseline measures of 
subdimensions; what 
instructional design methods, 
application of UDL, alignment 
with standards, course syllabi 
and course materials and 
content were employed prior to 
COVID-19? 


Challenges 


Within each 
subdimension, what 
challenges remained, 
arose, or became more 
amplified as a result of 
COVID-19? 


Qualitative; survey 
analysis and policy 
analysis 


Data collected via 
survey; policies 


provided for analysis by 


NELC (if there has been 
policy change and/or 
new policy following 
COVID-19) 


Within each 
subdimension, what 
challenges remained, 
arose, or became more 
amplified as a result of 
COVID-19? 


During COVID-19 
Response 


Within each 
subdimension, how 
has leadership 
responded to COVID- 
19? 


Qualitative; survey 
analysis and policy 
analysis 


Data collected via 
survey; policies 
provided for analysis 
by NELC (if there has 
been policy change 
and/or new policy 
following COVID-19) 


Within each 
subdimension, what 
has the response 


related to curriculum 


design and 
instruction been? 


Impact 


Within each 
subdimension, what 
impact has occurred 
due to leadership 
responses to COVID- 
19? 


Qualitative; survey 
analysis and policy 
analysis 


Data collected via 
survey; policies 
provided for analysis 
by NELC (if there has 
been policy change 
and/or new policy 
following COVID-19) 


Within each 
subdimension, what 
has the impact of 
COVID-19 on 
curriculum and 
instruction been? 


Post COVID-19 
Recommendations 


Recommendations 
resulting from 
analyses, both by 
overall dimension 
and subdimension. 


Qualitative; survey 
analysis and policy 
analysis 


Data collected via 
survey; policies 
provided for 
analysis by NELC 


Recommendations 
resulting from 
analyses, both by 
overall dimension 
and subdimension. 
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3 


Online 
Teaching & 
Learning 


Methodology 


Instruments 
(Data Source) 


Qualitative; survey analysis and 
document review 


Data collected via survey; 
documents for review requested 
as part of survey instrument 


Qualitative; survey 
analysis and document 
review 


Data collected via 
survey; documents for 
review requested as part 
of survey instrument 


Qualitative; survey 
analysis and 
document review 


Data collected via 
survey; documents for 
review requested as 
part of survey 
instrument 


Qualitative; survey 
analysis and document 
review 


Data collected via 
survey; documents for 
review requested as 
part of survey 
instrument 


Instructional Design Methods & UDL, Alignment with Standards, Course Syllabi, Course Materials & Content, Innovation 


Objectives 


Methodology 


Instruments 
(Data Source) 


Baseline measures of 
subdimensions; what methods 
of online course communication, 
engagement, expectation 
setting, outcomes, course 
interaction, feedback, and 
resources were employed prior 
to COVID-19? 

Mixed methods; survey analysis, 
document review, data analysis 


Data collected via survey; 
documents for review and data 
for analysis requested as part of 
survey instrument 


Within each 
subdimension, what 
challenges remained, 
arose, or became more 
amplified as a result of 
COVID-19? 


Mixed methods; survey 
analysis, document 
review, data analysis 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Within each 
subdimension, what 
has the response 
related to online 
teaching and learning 
been? 


Mixed methods; 
survey analysis, 
document review, 
data analysis 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Within each 
subdimension, what 
impact has occurred 
due to online teaching 
and learning responses 
to COVID-19? 


Mixed methods; survey 
analysis, document 
review, data analysis 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Communication, Engagement, Expectation Setting, Outcomes, Course Interaction, Feedback, Resources, Innovation 


Qualitative; survey 
analysis and 
document review 


Data collected via 
survey; documents 
for review 
requested as part 
of survey 
instrument 


Recommendations 
resulting from 
analyses, both by 
overall dimension 
and subdimension. 


Mixed methods; 
survey analysis, 
document review, 
data analysis 


Data collected via 
survey; documents 
for review and data 
for analysis 
requested as part 
of survey 
instrument 
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Objectives 


Methodology 


Instruments 
(Data Source) 


Baseline measures of 
subdimensions; what 
assessment strategies, 
processes, methodologies, and 
types were employed prior to 
COVID-19? 

Mixed methods; survey analysis, 
document review, data analysis 


Data collected via survey; 
documents for review and data 
for analysis requested as part of 
survey instrument 


Within each 
subdimension, what 
challenges remained, 
arose, or became more 
amplified as a result of 
COVID-19? 

Mixed methods; survey 
analysis, document 
review, data analysis 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Within each 
subdimension, what 
has the response 
related to assessment 
been? 


Mixed methods; 
survey analysis, 
document review, 
data analysis 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Assessment Strategies, Assessment Processes, Assessment Methodology, Assessment Types, Innovation 


Objectives 


Baseline measures of 
subdimensions; what was 
present and used regarding 
operability, centralized online 
education infrastructure, 


Within each 
subdimension, what 
challenges remained, 
arose, or became more 


Within each 
subdimension, what 
has the response 
related to technology 
been? 


Within each 
subdimension, what 
impact has occurred 
due to online 
assessment responses 
to COVID-19? 

Mixed methods; survey 
analysis, document 
review, data analysis 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Within each 
subdimension, what 
impact has occurred 
due to technology 


Recommendations 
resulting from 
analyses, both by 
overall dimension 
and subdimension. 


Mixed methods; 
survey analysis, 
document review, 
data analysis 


Data collected via 
survey; documents 
for review and data 
for analysis 
requested as part 
of survey 
instrument 


Recommendations 
resulting from 
analyses, both by 
overall dimension 
and subdimension. 
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Student 
Support 


modality, security, ITSM 
Compliance, internet access, 
reliability, and coverage prior to 
COVID-19? 


Methodology Qualitative; survey analysis 


Instruments 
(Data Source) 


Data collected via survey 


amplified as a result of 
COVID-19? 


Qualitative; survey 
analysis 


Data collected via 
survey 


Qualitative; survey 
analysis 


Data collected via 
survey 


responses to COVID- 
19? 


Qualitative; survey 
analysis 


Data collected via 
survey 


Qualitative; survey 
analysis 


Data collected via 
survey 


Operability, Centralized Online Education Infrastructure (SSO, LMS, etc.), Modality, Security, ITSM Compliance, Internet Access, Reliability, Coverage, 


Innovation 

Baseline measures of 
subdimensions; what was the 
availability and use of student 
orientation and support services, 
equity, accessibility, and 
compliance prior to COVID-19? 


Objectives 


Methodology Mixed methods; survey analysis, 
document review, and data 


analysis 


Within each 
subdimension, what 
challenges remained, 
arose, or became more 
amplified as a result of 
COVID-19? 


Mixed methods; survey 
analysis, document 
review, and data 
analysis 


Within each 
subdimension, what 
has the response 
related to student 
support been? 


Mixed methods; 
survey analysis, 
document review, 
and data analysis 


Within each 
subdimension, what 
impact has occurred 
due to online student 
support needs and 
responses to COVID- 
19? 

Mixed methods; survey 
analysis, document 
review, and data 
analysis 


Recommendations 
resulting from 
analyses, both by 
overall dimension 
and subdimension. 


Mixed methods; 
survey analysis, 
document review, 
and data analysis 
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Instruments 
(Data Source) 


Data collected via survey; 
documents for review and data 
for analysis requested as part of 
survey instrument 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Student Orientation & Support, Equity, Accessibility, Compliance Standards, Innovation 


Objectives 


Methodology 


Instruments 
(Data Source) 


Baseline measures of 
subdimensions; what was the 
availability and use of technical 
assistance, professional 
development, orientation, 
mentoring, and/or other support 
prior to COVID-19? 

Mixed methods; survey analysis 
and data analysis 


Data collected via survey; 
documents for review and data 
for analysis requested as part of 
survey instrument 


Within each 
subdimension, what 
challenges remained, 
arose, or became more 
amplified as a result of 
COVID-19? 


Mixed methods; survey 
analysis and data 
analysis 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Within each 
subdimension, what 
has the response 
related to training 
and support been? 


Mixed methods; 
survey analysis and 
data analysis 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Technical Assistance, Professional Development, Orientation, Mentoring, Innovation 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Within each 
subdimension, what 
impact has occurred 
due to training and 
support responses to 
COVID-19? 


Mixed methods; survey 
analysis and data 
analysis 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Data collected via 
survey; documents 
for review and data 
for analysis 
requested as part 
of survey 
instrument 


Recommendations 
resulting from 
analyses, both by 
overall dimension 
and subdimension. 


Mixed methods; 
survey analysis and 
data analysis 


Data collected via 
survey; documents 
for review and data 
for analysis 
requested as part 
of survey 
instrument 


Evaluation& Objectives 
Continuous 
Improvement 
Methodology 
Instruments 


(Data Source) 


Baseline measures of each 
subdimension; levels of student, 
faculty, and staff satisfaction, 
evaluation of course outcomes, 
and program quality prior to 
COVID-19. 


Mixed methods; survey analysis, 
document review, data analysis 


Data collected via survey; 
documents for review and data 
for analysis requested as part of 
survey instrument 


Within each 
subdimension, what 
challenges remained, 
arose, or became more 
amplified as a result of 
COVID-19? 


Mixed methods; survey 
analysis, document 
review, data analysis 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Within each 
subdimension, what 
has the response 
related to evaluation 
and continuous 
improvement been? 


Mixed methods; 
survey analysis, 
document review, 
data analysis 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Within each 
subdimension, what 
impact has occurred 
due to evaluation and 
continuous 
improvement 
responses to COVID- 
19? 

Mixed methods; survey 
analysis, document 
review, data analysis 


Data collected via 
survey; documents for 
review and data for 
analysis requested as 
part of survey 
instrument 


Student Satisfaction, Faculty Satisfaction, Staff Satisfaction, Evaluation of Course Outcomes and Program Quality, Innovation 


Recommendations 
resulting from 
analyses, both by 
overall dimension 
and subdimension. 


Mixed methods; 
survey analysis, 
document review, 
data analysis 


Data collected via 
survey; documents 
for review and data 
for analysis 
requested as part of 
survey instrument 
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Research Findings 


Subdimensions 


_ 
Strategies 


Process 


Before 


During Covid 19 


Post Covid 19 


Appropriate governance 
structures are in place to 
support online education 


Institutions have developed 
and articulated institutional 
strategies for online 
learning while aligning with 
existing university- and 
program-level plans 


Institutions have policies 
that encourage the 
development of online 
courses and programs 


Institutional structure 
provides centralized support 
for online program 
development, faculty within 
online programs, and 
students enrolled in online 
programs 


Involve faculty in governance 
and related decision-making 


Communicate with faculty 
and involve them as 
appropriate regarding 
strategic planning 


Ensure comprehensive and 
clear policies support online 
education; communicate with 
and involve faculty as 
appropriate 


Even implementation of 
processes that are effective 
for online programs 


No explicit actions 
described 


No explicit actions 
described 


No explicit actions 
described 


No explicit actions 
described 


Consistent satisfaction pre- 
and post-COVID, though very 
satisfied decreased slightly 
and neutral and dissatisfied 
responses increased slightly 


Majority satisfaction with 
strategies, though slight 
decreases occurred following 
COVID 


Majority satisfaction with 
policies, though lower faculty 
agreement than 
administrator/staff 


Majority satisfaction with 
processes 


Provide sufficient opportunities 
for faculty to be involved in 
decisions related to online 
learning 


Include online learning in 
strategic plans at the 
institutional level in order to 
appropriately align resources; 
Develop a strategic plan for 
online learning 


Develop national policies on 
the use of online education in 
higher education that can 
provide universities with 
guidelines and expectations for 
a quality learning experience 


Ensure that processes (e.g., 
admissions, registrar requests, 
testing, etc.) work with students 
at a distance 


Resource 
Allocation 


Periodic Review 
& Updating 


Innovation 


Training and development 
or faculty development 
programming are in place to 
provide knowledge, skills, 
and abilities for the design 
and teaching of online 
courses 


Institutions have 
governance structures to 
enable systematic and 
continuous improvement 
related to the administration 
of online education 


Majority satisfaction with 
innovation in leadership 


Development and 
implementation of processes 
on planning and allocating 
resources for online learning, 
incentives for related course 
development and design, 
consideration of the 
positional parity of 
instructional designers 


Ongoing implementation of 
review and updating of 
processes 


Allocation of financial support 
and incentives for innovation; 
inclusion of innovation in 
teaching and learning as part 
of evaluation and promotion 
policies 


implementation of 
resource allocations; 
some actions 
localized institutionally 
to address challenges 


No explicit actions 
described; increase in 
satisfaction indicates 
increased visibility 
and/or adoption of 
such processes 


No explicit actions 
described, though 
some localized 
actions may have 
been implemented as 
satisfaction increased 
after the pandemic 


Consistency in overall pre- 
and post-COVID satisfaction; 
areas of challenge identified 


Decrease in satisfied and 
neutral responses and 
increase in very satisfied 
responses 


Increase in satisfaction post- 
COVID; discrepancies remain 
between administrators/staff 
and faculty 


Provide incentives to faculty to 
participate in professional 
development and create online 
courses; Evaluate internet 
availability throughout the 
Kingdom of Saudi Arabia to 
determine opportunities to 
improve access; Prioritize 
access to technology, 
technology support, and 
technology training to better 
support students 


Conduct regular institutional 
reviews of online learning 
courses and programs; 
Conduct regular and 
substantive reviews of online 
learning at the National level 


Rewards, recognition or other 
incentives should be provided 
to encourage innovation 
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Curriculum 
Design & 
Planning 


Instructional 
Design Methods 
& UDL 


Alignment with 
Standards 


Course Syllabi 


Processes are in place to 
ensure effective 
development and 
implementation of 
curriculum and design 


Courses are aligned with 
best practices and quality 
standards for design, 
development, and delivery 


Majority agreement that 
syllabi contain essential 
course and content 
information 


Increase human resources 
for better quality assurance; 
faculty involvement in 
curriculum design and 
development; training in 
instructional design; 
inconsistent student 
experiences 


Ensure that alignment is 
accurate and supports 
students’ ability to develop 
necessary knowledge and 
skills to achieve learning 
outcomes 


Include expectations for 
participation in syllabi 


Independent and 
localized efforts at 
institutional levels are 
indicated 


No explicit actions 
described 


No explicit actions 
described 


Consistency in satisfaction 
pre- and post-COVID 


Slight decreases in 
satisfaction following COVID- 
19, indicating that the 
pandemic illuminated or 
resulted in additional 
alignment needs 


Slight decrease in satisfaction 
post-COVID, potentially due to 
the rapid shift to remote 
teaching 


Provide regular opportunities 
for faculty input on curriculum 
design and planning decisions, 
and clearly communicate with 
faculty regarding such 
decisions to ensure buy-in 
across the institution(s) and 
effective implementation of 
decisions; Develop and 
regularly offer professional 
development and training on 
instructional design and UDL, 
and provide ongoing 
instructional design support to 
faculty; Include instructional 
design in quality review 
processes and whenever 
possible, employ standardized 
practices 


Ensure alignment with 
standards through quality 
review, and collect student 
feedback and measure their 
learning outcomes to improve 
and monitor effectiveness 


Develop and implement clear 
syllabi components, including 
clearly labeled course 
objectives, learning outcomes, 
assessment methods, 
descriptions of content and 
activities, and expectations; 
Ensure that syllabi are easy to 
locate and use in a consistent 
course design practice 


Materials are accessible, Ensure sufficient breadth, No explicit actions are | Slight decrease in satisfaction |Regularly review course 
Materials & easy to use, and may be depth, and currency of described and increase in neutral materials and content for 
Content accessed on multiple instructional materials; responses following COVID __|sufficient coverage and 
devices; a variety of include current content such currency, and update as 
content/materials types as online resources and/or appropriate with current and 
appropriate for multiple videos media-rich content; Ensure 
learning objectives are academic rigor of content with 
present activities at the appropriate 
level of Bloom’s Taxonomy 


Innovation Institutions support Student No explicit actions Consistent responses pre- Provide opportunities for 
innovation in curriculum communication/feedback described and post-COVID student feedback regarding 
design and planning and/or recognition of innovative pedagogical and 

innovative planning and technological practices in 
practices curriculum design and planning 


Communication |Faculty are expected to Increase communications Independent faculty |Decrease in satisfaction with |Consider identifying and 
hold virtual office hours and |with students in the virtual changes to communication post-COVID __|defining a list of core definitions 
frequently communicate setting in ongoing, timely, communication types as they apply to the institution; 
with their students in effective, and appropriate and frequencies Consider what measurements 
various ways ways (if any) define these terms 


Engagement Course design promotes Ensure that students are No explicit actions Decrease in satisfaction post- | Provide interactive 

student engagement, academically challenged and |described COVID, potentially due to the |opportunities, both formal and 

including academic interact with each other and rapid shift to remote informal, for students to interact 

challenge and social with faculty instruction with one another and faculty, 

involvement with each other and ensure that academic 

and the instructor challenge is incorporated into 
these experiences as 
appropriate 


Expectation Courses contain clearly Ensure clear expectations No explicit actions Decrease in satisfaction and |Ensure that online courses 
Setting defined student activity and effective communication |described, though increase in neutral and include clearly defined student 
expectations of expectations to students __ | faculty clearly dissatisfied responses post- _| activity expectations, and that 
understand the COVID these expectations are 
importance of doing communicated to students; 
so Provide information on 
expectations for teacher 
responsiveness 


ONLINE LEARNING 
CONSORTIUM 


ORC! 


Outcomes 


Course 
Interaction 


Feedback 


Teaching and 
Learning 
Resources 


Innovation 


Mixed results regarding 
whether onsite and online 
outcomes are equivalent 


Student-to-student and 
faculty-to-student 
interactions are encouraged 
and facilitated 


Faculty and instructors 
provide timely and detailed 
feedback on assessments 
and student inquiries 


Administrators/staff and 
faculty indicate that course 
resources provide richness 
in learning materials and 
activities, support and 
instruction, instructor 
interactions, and tools and 
media 


Administrators/staff and 
faculty indicate that 
innovation is encouraged 
for online teaching and 
learning 


Ensure equivalence of 
outcomes regardless of 
modality; address perceived 
gaps in technology for 
teaching and assessing 
hands-on needs (such as 
medical or nursing courses) 


Ensure that technology 
availability and selection is 
sufficient to support quality 
interactions 


Clear communication of 
feedback is provided to and 
understood by students 


Resources are present and 
students recognize and 
utilize them 


Increase student awareness 
of innovation 


Independent and 
localized efforts at the 
institutional level are 
described 


Independent and 
localized efforts at the 
institutional level are 
described 


No explicit actions are 
described 


No explicit actions are 
described 


No explicit actions are 
described 


Decrease in satisfaction and 
increase in neutral and 
dissatisfied responses post- 
COVID 


Decrease in satisfaction and 
increase in neutral and 
dissatisfied responses post- 
COVID 


Decrease in satisfaction and 
increase in neutral and 
dissatisfied responses post- 
COVID 


Decrease in satisfaction and 
increase in neutral and 
dissatisfied responses post- 
COVID 


Decrease in satisfaction and 
increase in neutral and 
dissatisfied responses post- 
COVID 


Measure, review, and 
communicate student learning 
outcomes for online courses, 
and if appropriate and/or 
useful, include communicating 
comparisons to student 
learning outcomes for onsite 
courses; Provide rubrics for all 
assignments 


Establish the presence of the 
online educator 


Set expectations for feedback 
provision and methods, as well 
as when feedback will be 
provided and how it is related 
to course learning objectives; 
Implement student experience 
measurements within online 
learning experiences as a part 


of the design process 


Develop and implement 
evaluation procedures to 
gather information regarding 
student perceptions of course 
resources, and include this 
data in continual course 
improvement processes 


Provide communications to 
increase student awareness of 
innovations implemented in 
online teaching and learning, 
and provide opportunities for 
student feedback regarding 
innovative practices 


Assessment 
Strategies 


Assessment 
Processes 


Assessment 
Methods 


Assessment 
Types 


Assessment strategies 
encompass student 
readiness for online 
learning, piloting strategies 
that include digital and other 
pertinent literacies, and 
formative assessment 


Administrators/staff and 
faculty indicate that 
assessments are aligned to 
learning objectives, criteria, 
graduation, and 
expectations are 
communicated, and 
feedback is provided 


New assessment methods, 
a variety of assessment 
methods, the use of multiple 
and timely assessment 
activities, and the belief that 
course grades are not 
based solely on exams are 
present 


Opportunities are present to 
demonstrate proficiency in 
ways other than just tests 
and quizzes 


Communicate with students |No explicit actions are 
to identify assessments, their | described 

purposes, and learning 

outcomes 


Communicate with students 
regarding assessment 
processes and set 
expectations for feedback 


No explicit actions are 
described, though 
independent and 
localized efforts are 
indicated 


Increase opportunities for 
self- and peer-assessment 
and feedback 


Independent 
improvement of online 
assessments is 
indicated by faculty 


Clearly communicate No explicit actions are 
different types of assessment | described 

and their purposes to 

students 


Decrease in satisfaction and 
increase in neutral and 
dissatisfied responses post- 
COVID 


Decrease in satisfaction and 
increase in neutral and 
dissatisfied responses post- 
COVID 


Decrease in satisfaction and 
increase in neutral and 
dissatisfied responses post- 
COVID, potentially influenced 
by the rapid shift to remote 
instruction 


Decrease in satisfaction and 
increase in neutral and 
dissatisfied responses post- 
COVID 


Develop and implement 
course-level communications to 
students that identify 
assessments, their purposes, 
and the learning outcomes for 
each 


Ensure alignment of learning 
objectives and assessments 
through quality review 
processes; Align course design 
practices regarding 
assessment with best practices 
for the placement and use of 
communications and materials 
including criteria, assignments, 
expectations, and grading 
rubrics; Set expectations for 
feedback provision and 
methods, as well as when 
feedback will be provided and 
how it is connected to each 
assessment 


Provide faculty with 
professional development 
opportunities on the design and 
delivery of both peer- and self- 
assessment methods for 
students 


Provide opportunities to assess 
learning at higher levels of 
knowledge acquisition 
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Innovation 


Centralized 
Online 
Education 
Infrastructure 
(SSO, LMS, 
etc.,) 


Internet Access 
and Reliability 


ITSM 
Compliance 


Modality 


Innovation through new 
strategies, the inclusion of 
current and relevant 
literacies, the use of new 
assessment methods, and 
opportunities to 
demonstrate learning in a 
variety of ways is supported 


Moderate agreement that 
centralized support 
infrastructure is widely 
available, including support 
for building and maintaining 
the technical infrastructure 


Mixed results regarding 
whether technology and 
internet access was 

available and/or reliable 


An ITSM plan is in place to 
design, delivery, operate, 
and control information 
technology in support of 
online program 
development 


Online courses are 
delivered with an effective 
mix of synchronous or 
asynchronous delivery, 


Provide professional 
development on innovative 
assessment 


Ensure equitable availability 
of centralized support 
infrastructure 


Assess needs and address 
equity of access and 
reliability 


Determine priorities for ITSM 
services and address 
infrastructure and support 
needs 


Review modality decisions 
and communicate to 
students about modality and 
learning outcomes 


No explicit actions are 
described 


No explicit actions are 
described; increased 
use of and exposure 
to online education 
infrastructure may 
have resulted in 
increased satisfaction 
following the 
pandemic 


No explicit actions are 
described, though 
there is an 
understanding that 
improvements are 
needed 


Some localized 
actions are indicated 


No explicit actions are 
described 


Decrease in satisfaction and 
increase in neutral and 
dissatisfied responses post- 
COVID 


Consistency in satisfaction 
pre- and post-COVID overall, 
though very satisfied 
responses increased and 
satisfied responses decreased 


Consistent and positive rates 
of satisfaction pre- and post- 
COVID, though uneven 
access and reliability are 
repeatedly described 


Consistent and positive rates 
of satisfaction are present 
both pre- and post-COVID 


Consistent and positive rates 
of satisfaction are present 
both pre- and post-COVID 


Innovation in assessment 
strategies, processes, 
methods, and types is 
encouraged and supported, 
and related professional 
development on design, 
development, and delivery is 
provided 


Evaluate technology 
infrastructure to ensure that it is 
sufficient to meet current and 
upcoming demands, and create 
and implement improvement 
plans as appropriate 


Alternatives should be 
considered such as a reduction 
in the number of synchronous 
course meetings, or technology 
and faculty development 
working together to support 
faculty in offering different, 
flexible access options that do 
not rely heavily on immediately 
accessible technology 


Create faculty development 
and support materials around 
how the academic technologies 
offered by IT work, and 
incorporate these services 
purposely into overall ITSM 
structures 


Evaluate student experiences, 
needs, and perceptions 
regarding synchronous and 
asynchronous learning in 
online courses and incorporate 


depending on learning findings into course design and 
outcomes delivery 


Operability and |The Learning Management |Ensure that operability and _|No explicit actions are |Consistent and positive rates |Operability and security should 
Security System has an established |security are regularly described of satisfaction both pre-and _|be regularly monitored and 
interoperability protocol reviewed and updated for post-COVID functionality maintained and 
established so that functionality updated as needed 
embedded applications 
provide a seamless 
experience; Digital security 
measures are in place to 
ensure the integrity and 
validity of information 
including identity access 


Technology Plan | There is a document Increase communication and | No explicit actions are | Majority agreement that Administrators/staff and faculty 
technology strategic plan in | visibility of how technology |described technology plans are in place |receive communications 
place to ensure quality plans are related to online regarding technology plans, 
standards in online learning |learning including clear indicators of 
how the plan is connected to, 
and supports, online education 


Coverage Sufficient coverage and Monitor coverage and make _ |No explicit actions are |Consistent and positive rates |Continual monitoring of 
ability to access items improvements as needed described of satisfaction both pre-and _| coverage and access to online 
needed for online learning post-COVID learning needs for 
are present administrators/staff, faculty, 
and students 


Innovation Administrators/staff and Increase communication and |No explicit actions are |Decrease in satisfied and Provide opportunities for 
faculty indicate agreement |understanding of technology |described increase in neutral responses |student feedback regarding 
that their institutions pilot, innovation for students post-COVID innovative technology 
support, and encourage 
innovative technologies 


Student Institutions provide an Evaluate needs and Independent faculty | Overall consistency in positive | Ensure availability of 

Orientation & orientation to course design |solutions for social-emotional |and localized efforts at|rates of satisfaction; decrease |orientations and student 

Support and technologies, support institutional levels are |in very satisfied an increase in |resources targeted to support 
information on university indicated neutral responses post- students and faculty in online 
technology and academic COVID learning environments; Provide 
support resources, and student support equal to those 
access to personnel that provided to face-to-face 
provide assistance students 


ORC! 


Training & 
Support 


ONLINE LEARNING™ 
CONSORTIUM 


Accessibility 


Compliance 


Standards 


Innovation 


Technical 
Assistance 


Professional 
Development 


Survey data indicates 
agreement that equitable 
access to resources, 
technology, course 
materials, and support is 
provided. Interviews present 
more mixed indications of 
equity. 


Online courses demonstrate 
compliance with 
accessibility standards 


Online courses meet 
compliance with 
accessibility standards 


Institutions support students 
in innovative ways 


Faculty and instructors 
receive sufficient technical 
assistance for course 
development and online 
teaching 


Administrators/staff and 
faculty agree that they had 
sufficient training and 
support 


Evaluate and ensure 
equitable access to 
necessary course items and 
supports 


Availability of tools and other 
resources to assist students 
with disabilities 


Consider economic 
competitiveness and online 
learning as compliance goals 
are developed and 
implemented 


Increase support for 
innovation at private 
institutions 


Address uneven provision of 
technical assistance options 
for students 


Support faculty in available 
time for professional 
development, establish clear 
engagement and expectation 
standards 


Localized actions at 
institutional levels are 
indicated; it is possible 
that onsite equity was 
satisfactory and the 
pandemic resulted in 
an increase in need 


No explicit actions are 
described 


No explicit actions are 
described 


Localized actions at 
an institutional level 
are indicated 


Independent actions 
by instructors are 
indicated 


Localized actions at 
institutional levels are 
indicate 


Decrease in satisfied and 
increase in neutral and 
dissatisfied responses post- 
COVID 


Decrease in satisfied and 
increase in neutral and 
dissatisfied responses post- 
COVID 


Decrease in satisfied and 
increase in neutral and 
dissatisfied responses post- 
COVID 


Decrease in satisfied and 
increase in neutral and 
dissatisfied responses post- 
COVID 


Consistent and positive 
satisfaction pre- and post- 
COVID 


Consistent and positive 
satisfaction pre- and post- 
COVID 


Ensure availability and 
communication of learning and 
technology requirements as 
well as available assistance to 
support faculty and students in 
online learning environments 


Ensure appropriate 
accessibility needs are in place 
through investigation of current 
practices 


Include all stakeholders 
including administrators/staff, 
faculty, Instructional Designers, 
and educational technology 
vendors in the implementation 
of compliance goals 


Invest in educational 
technology tools and research 
to stimulate innovative online 
educational practices and 
support services 


Administrators, staff, and 
faculty generally felt that there 
was sufficient technical 
assistance, but students 
sampled for this study tended 
to have a more negative 
response to technical issues 
and support; Provide adequate 
technical assistance for faculty 


Increase communication 
between those stakeholder 
groups to improve rating by 
faculty; Move long synchronous 
trainings into modular, 
asynchronous formats; Provide 
regular opportunities for 


professional development 
specific to online learning 


Orientation Faculty and instructors Evaluate reasons for Localized actions at | Consistent and positive The Ministry or NELC could 
receive sufficient satisfaction/dissatisfaction institutional levels are |satisfaction pre- and post- work with some institutions to 
technology and instructional |with orientation and address _|indicated COVID identify these technologies and 
design orientation them as appropriate develop and share common 

orientations that could be used 
by anyone within the Kingdom 


Mentoring Faculty and instructors Evaluate access to and No explicit actions are |Consistent and positive A virtual mentoring program 
receive sufficient peer ability for participation in peer} described satisfaction both pre- and coordinated centrally could 
mentoring resources mentoring resources post-COVID, though satisfied | provide opportunities for 


responses decreased and greater connection and support 
neutral responses increased 
following the pandemic 


Innovation Faculty and instructors are |Support both organized and |Independent and Consistent and positive Support the awareness and 
informed about emerging independent efforts toward |localized actions are _|satisfaction pre- and post- use of new technologies is 
technology and new tools _{innovation indicated COVID through a faculty learning 

community program. These 
communities can take place 
organically, led by a university 
staff member or faculty 
member, or can be coordinated 
centrally through divisions of 
academic/faculty affairs or 
teaching and learning centers 
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Evaluation of 
Course 
Outcomes and 
Program Quality 


Student, Faculty, 
and Staff 
Satisfaction 


Innovation 


Localized actions at 
institutional levels are 
indicated 


Procedures and policies for | Consider implementing 
evaluating and improving and/or increasing faculty 
programs and course-level |peer evaluation 
alignment and evaluation 

are in place 


Increase communication of 
and opportunities for 
students to provide feedback 
on course improvements and 
evaluations of the quality of 
instruction and online 
materials 


Stakeholder assessments 
for students, faculty, and 
staff are in place 


No explicit actions are 
described 


Continual implementation of 
appropriate assessment 
methods for online 
education including 
development, 
implementation, and 
communication regarding 
innovative analysis methods 
used to inform 
improvements are present 


Evaluate and continually 
improve innovation as 
appropriate to the current 
environment 


No explicit actions are 
described 


Consistent and positive 
satisfaction both pre- and 
post-COVID, though satisfied 
responses decreased and 
neutral responses increased 
following the pandemic 


Overall consistency and 
positivity in satisfaction both 
pre- and post-COVID across 
all three groups, though small 
shifts occurred in the 
distribution of responses 


Consistent and positive 
satisfaction both pre- and 
post-COVID, though satisfied 
responses decreased and 
neutral responses increased 
following the pandemic 


Ensure adherence to 
established processes, 
procedures, and policies on 
course and program 
evaluation, including regular 
review and updating; Provide 
communication and 
transparency to faculty, and 
involve faculty when possible in 
initiatives regarding analytics 
for programmatic decisions; 
Develop and implement 
practices and expectations 
regarding faculty peer 
evaluation, including explicit 
processes and procedures as 
well as appropriate 
professional development on 
the development, 
implementation, and use of 
such practices 


Communications are provided 
to faculty, students, and staff 
regarding evaluation 
opportunities and their 
purpose, including information 
on the use of such feed for 
course and program 
improvements 


Incorporate best practices 


Appendix III: Terminology 


This appendix contains the most common current terms used to describe various iterations of learning 
via remote, distance, and online means. Below are commonly accepted terminology definitions, but it 
should be noted that they often overlap and even contradict one another. Guidelines for selecting terms 
to be used institutionally or ona larger scale, such as system- or kingdom-wide can be found in the 
Terminology section of Appendix VI: Literature Review. 


= Emergency Remote Teaching: A temporary shift of instructional delivery from its original 
modality to one that fits the current situation due to crisis situations. 

= Online Education: Courses are designed, developed, and implemented in a fully online 
environment. Courses may be synchronous, asynchronous, or a combination. 

" Distance Education: Teaching and learning is provided at a distance and may be offered 
through various means such as online, video, correspondence, etc. 

= eLearning: Teaching and learning is provided through the use of electronic technologies. 

" Hybrid/Blended Courses: Courses are provided in a format blending multiple modalities. Most 
often, some face-to-face components are combined with online and/or distance education. 

= HyFlex Courses: Courses are designed and offered in a flexible format allowing for face-to-face, 
online, or hybrid/blended modalities that may be selected based on need or preference. 


Embedded in these terms used to describe overall variations in online learning are several terms 
describing common practices. 

= Synchronous: The learner and instructor are online at the same time for synchronous 
instructional delivery, course activities, and/or interaction and engagement. 

= Asynchronous: Content delivery, course activities, and opportunities for engagement and 
interaction may be completed independently or at different times for each instructor and 
learner. 

=" Online Collaborative Learning: The use of synchronous, asynchronous, or mixed methods for 
course delivery and completion 

=" Computer-based training: The provision of modular online learning, often in a self-paced 
format. This type of online learning is typically used in continuing education or professional 
development. 

« Digital Education, or Technology-Enhanced Learning: The use of digital technologies to 
provide education, including systems (i.e. Learning Management Systems), tools (i.e. 
applications for communication or learning activities), or course artifacts (i.e. digital activities 
and interactions). 


There are also a number of course-level terms that are essential for development and delivery of online 
learning. 
« Instructional Design: The systematic process of designing, developing, and delivering effective 
instruction through effective structure and presentation of learning materials and content, 
course activities and interactions, and assessment. 
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Universal Design for Learning (UDL): A framework to improve and optimize teaching and 
learning for all individuals. 

Learning Outcomes: What the learners will demonstrably know and/or be able to do at the end 
of a course. 

Learning Objectives: What the learners will demonstrably know and/or be able to do as a result 
of individual units or activities. 

Course Materials and Content: Learning content and supporting materials designed to support 
learning outcomes and objectives. 

Communication: Provision and exchange of information, including email, discussions, 
synchronous meetings, documents, etc. in a course. 

Engagement: The level of student thinking, communication, and interaction with course 
content, materials, activities, and participants. 

Course interaction: Active student engagement with course components and participations. 
Generally considered in three forms. 

e Student-Content Interaction: Active student engagement with learning content, 
materials, activities, and assessments. 

e Student-Student Interaction: Active student engagement with other students, 
including video or discussion, group projects, collaborative work, etc. 

e Student-Instructor Interaction: Active student engagement with instructors, 
including communications, discussions, feedback, participation through course 
activities, etc. 

Feedback: Communication about performance (or experience) that is provided and 
subsequently used to increase understanding and/or improve outcomes. 
Assessment: The evaluation, measurement, and documentation of learning and/or 
achievement of outcomes and objectives. 


Appendix IV: NELC Future Action Framework for Higher Education 


Leadership 


Strategies 


Governance 


Online Learning 
Governance 


One or more items 
related to online 
learning within the 
institutional 
strategic plan 
Online learning 
strategic plan 


Provide 
opportunity for 
faculty to be 
involved in 
decision making 
related to online 
learning 


Prioritize online 
learning 


Provide specific 
and actionable 
items related to 
online learning 


Establish a 
committee or 
advisory group for 
online learning 
made up of 
faculty or include 
faculty in an 
existing 
committee 


Prioritize online 
learning by 
including it in the 
strategic plan 


Develop a 
strategic plan for 
online learning 


- Form committee structure 
including mission and objectives 
- Identify appropriate faculty to 
serve on committee (through 
election, invitation, etc.) - 
Include administrators on 
committee as appropriate - 
Identify opportunities for faculty 
to make decisions on curriculum 
and other needs for online 
learning - Define how decisions 
will be made by the committee 
Review existing strategic plan 
and revise to include online 
learning where appropriate 


- Identify committee made up of 
representatives from throughout 
the institution to serve on 
committee to develop 

- Write strategic plan 

- Align resources to strategic 
plan 

- Evaluate effectiveness of 
strategic plan 


Active engagement 
of faculty in online 
learning decision 
making 


Commitment to 
online learning 


Sustain online 
learning 


- Faculty 
involvement in 
decision making - 
Time to make 
decisions - 
Success of 
decisions made 
over longer term 


Inclusion of 
online learning in 
institutional 
strategic plan 


-Development of 
strategic plan 
-Alignment of 
resources to 
strategic plan 


Audie 


Administrators 
at each higher 
education 
institution 
(specifically 
should be at the 
level that has 
oversight for 
faculty) 


Institutional 
leadership 


Institutional 
Leadership 
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Policies Policies specific to 
online learning 

Process Processes to 
support online 
learners 

Resource Rewards, 

Allocation recognition or 


other incentives 


Adequate internet 
access 


Establish 
standards of 
quality 


Processes to 
support virtual 
learning 


Encourage faculty 
to develop online 
courses 


Increase 
accessibility to 
online learning 


- Develop 
appropriate 
policies related to 
online learning to 
encourage 
effective 
implementation 

- Develop 
standards of 
quality for online 
learning 
curriculum and 
programs 
Processes that 
meet the needs of 
online learning 


Provide 
incentives to 
faculty to 
participate in 
professional 
development in 
order to develop 
online courses 


Provide adequate 
internet access 
throughout the 
Kingdom of Saudi 
Arabia (rural, 


- Write policies for online 
learning at the course and 
program level (e.g., 
requirements to be able to offer 
online courses/programs, 
training requirements for 
faculty/staff, student services 
that need to be available, 
technical requirements, etc.) - 
Share policies with institutions 
of higher education - Review 
implementation of online 
learning against policies 
Implement appropriate 
processes to support online 
learners 


Determine what types of 
incentives will be used 
institutionally (e.g., financial, 
additional resources, 
recognition, etc.) - Create 
guidelines to receive incentive 
(e.g., attend certain number of 
workshops, develop first online 


course, teach online course, etc.) 


- Determine weak points for 
internet access throughout the 
country 

- Identify locations to install or 
upgrade internet 


Effective 
implementation of 
online learning 


- Review existing 
processes and 
modify for online 
learners 

- Identify and 
develop additional 
processes needed 
to support online 
learners 

Faculty 
commitment 


Learners able to 
access online 
courses 


- Policies written 
and shared - 
Compliance with 
policies 


Processes written 
and implemented 


- Increase in 
number of faculty 
qualified to teach 
online - Increase 
in number of 
online courses 
offered 


- Increase in 
number of 
students able to 
enrollin online 
learning 


National 
eLearning 
Center 


Administration, 
faculty 


Administrators 


Government 
(possibly 
regional) 


Periodic 
Review and 
Updating 


Technology, 
Technology 
Support and 
Technology 
Training 


Policies, processes 
and procedures 
specific to the 
regular review of 
online learning 
courses and 
programs 


Improve use of 
technology 


Development of 
policies, 
processes and 
procedures to 
review and 
provide feedback 
on online learning 
courses and 
programs 


suburban, and 
urban) 


Allocate 
appropriate 
resources to 
support the use of 
technology for 
online learning 


Maintain and 
regularly update 
the quality of 
online courses 
and programs 


- Review technology being used 
to support online learning for 
effectiveness - Provide 
technology support for students 
and faculty that encounter 
issues with being able to access 
their online course - Provide 
professional development or 
workshops to students and 
faculty to help more effectively 
use educational technology 


- Develop and implement 
policies, processes and 
procedures specific to the 
regular review of online learning 
courses and programs 

- Use standardized rubrics and 
scorecards (Online Learning 
Consortium, Quality Matters) as 
part of the process to review 
against quality standards 


- Conduct regular reviews as part 


of continuous improvement 
process 


Learners and 
faculty able to 
more effectively 
engage with online 
classroom 


Online learning 
courses and 
programs continue 
to meet best 
practices 


- Increased use of | Administrators 
technology to 

support online 

learning - 

Increased access 

to support for 

technology issues 

- Increased 

training for use of 

required 

technology - 

Increased student 

and faculty 

satisfaction with 

online learning 

Administrators, 
Faculty 


- Online learning 
courses and 
programs 
reviewed per 
policy 

- Online learning 
courses and 
programs 
continuously 
updated 
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and procedures 
specific to the 
regular review of 
online learning 
programs 


Innovation Rewards, 
recognition or 


other incentives 


Instructional 
Design 
Methods & UDL 


Curriculum 
Design & 
Planning 


input on 


and planning 
decisions 


- Communication 


regarding such 
decisions 
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Policies, processes 


- Opportunities for 


curriculum design 


Development of 
policies, 
processes and 
procedures to 
review and 
provide feedback 
to institutions on 
online learning 
courses and 
programs 
Encourage faculty 
to be more 
pedagogically or 
technologically 
innovative 


- Create 
opportunities for 
faculty 
participation in 
decision-making, 
such as 
committee 
involvement and 
soliciting written 
input and/or 
feedback - 
Regularly 
communicate 
with faculty 


regarding 


Ensure the quality 
of online learning 
programs offered 
at institutions 


Provide 
incentives to 
faculty to 
participate in 
professional 
development in 
order to develop 
online courses 


To utilize faculty 
expertise by 
including them in 
curriculum design 
and planning 
decisions and to 
increase the 
awareness and 
understanding of 
such decisions 


- Develop and implement 
policies, processes and 
procedures specific to the 


regular review of online learning 


programs 

- Provide feedback on quality of 
program compared to industry 
standards 


- Determine what types of 
incentives will be used 
institutionally (e.g., financial, 
additional resources, 
recognition, etc.) - Create 
guidelines to receive incentive 
(e.g., attend certain number of 
workshops, develop first online 
course, teach online course, etc.) 


Create opportunities and 
communications plans for 
faculty involvement in 
curriculum design and planning 
decisions 


Quality online 
learning programs 
offered within the 
Kingdom of Saudi 
Arabia 


Faculty 
commitment to 
innovation 


- Presence and 
awareness of 
opportunities for 
faculty 
participation - 
Regular 
communications to 
faculty regarding 
decisions 


National 
eLearning 


- Policies, 
processes and 
procedures 
developed and 
shared with 
institutions 


Center 


- Regular reviews 
of online learning 
programs 
conducted 

- Increase in Administrators 
number of faculty 
engaging in 
pedagogical 
innovation - 
Increase in 
number of faculty 
engaging in 
technological 
innovation 

- Faculty are Faculty 
aware of and 

participate in 

curriculum design 

and planning 
decision-making 
opportunities - 
Communications 

are created and 

sent regularly, 

and faculty 

indicate 

awareness and 
understanding of 


- Professional 
development and 
training on 
instructional 
design and UDL - 
Ongoing 
instructional 
design support for 
faculty 


Standardized 
course design 
practices 


curriculum design 
and planning 
decisions 


- Develop and 
regularly offer 
professional 
development and 
training focused 
on instructional 
design and UDL - 
Develop ongoing 
instructional 
design support 
resources for 
faculty 


- Develop and 
implement 
standardized 
instructional 
design practices - 
Include 
instructional 
design in quality 
reviews 


To increase 
faculty 
knowledge, skill, 
and support 
resources on 
instructional 
design and UDL 


To increase 
course 
consistency and 
coherence 
through the use of 
standardized 
practices and 
quality review 


Develop and regularly offer 
professional development and 
training on instructional design 
and UDL, and ensure that faculty 
are aware and able to access 
both training and support 
options 


- Develop standardized 
instructional design practices - 
Ensure that faculty understand 
and implement these practices - 
Include these practices in quality 
course reviews 


- Presence of 
professional 
development and 
support 
opportunities - 
Awareness of 
professional 
development and 
support 
opportunities 


- Standardized 
instructional 
design practices 
are developed - 
Standardized 
instructional 
design practices 
are implemented - 
Quality course 
reviews include 
instructional 
design 


decisions 


- Faculty 
participation in 
professional 
development 
opportunities - 
Faculty use of 
support resources 
- Evaluation of 
effectiveness 
based on course 
design outcomes 
and faculty 
feedback 

- Instructional 
design standards 
are evident in 
courses - Quality 
course reviews 
reflect adherence 
to standardized 
instructional 
design - Student 
feedback 
indicates an 
increase in 
consistent and 
coherent design 
across courses 


Faculty 


Faculty, 
Students 
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Alignment with 
Standards 


Ensure that 
courses are aligned 
with standards and 
that students are 
able to develop the 
knowledge and 
skills needed to 
achieve learning 
outcomes 


- Quality course 
review for 
alignment with 
standards - 
Collect student 
feedback on their 
ability to achieve 
learning 
outcomes - 
Regularly monitor 
student outcomes 
in courses 


To increase 
alignment to 
standards in 
courses to 
support 
successful 
student learning 
outcomes 


- Include alignment with 
standards in quality course 
reviews - Provide opportunities 
for student feedback regarding 
their ability to achieve learning 
outcomes - Measure student 
outcomes in courses for 
continuous improvement 


- Alignment with 
standards is 
reflected in quality 
course reviews - 
Students are able 
to provide 
feedback regarding 
their ability to 
achieve learning 
outcomes - 
Student outcomes 
in courses are 
measured and 
reviewed regularly 


- Alignment with 
standards is 
evident in quality 
course reviews - 
Students 
participate in 
feedback 
opportunities and 
indicate areas of 
success or 
challenge with 
their ability to 
achieve learning 
outcomes - 
Measurement and 
review of student 
outcomes in 
courses reflects 
improvement 
and/or is used for 
continuous 
improvement 


Faculty, 
Students 


Course Syllabi 


Course 
Materials & 
Content 


- Syllabi contain 
standard 
components that 
are clearly labeled 
and include course 
objectives, learning 
outcomes, 
assessment 
methods, 
descriptions of 
content and 
activities, and 
expectations - 
Syllabi are easy to 
locate and use with 
consistency in 
course design 


Ensure that course 
materials and 
content include 
sufficient coverage 
and are regularly 
updated with 
current and media- 
rich content 


Ensure that course 
content (activities, 
assignments) are at 
an appropriate 
level of academic 
rigor and include 


- Develop and 
implement 
standard syllabi 
components - 
Implement course 
design standards 
ensuring that 
syllabi are 
consistently easy 
to locate and use 


- Regularly review 
course materials 
and content - 
Regularly update 
course materials 
and content with 
current and 
media-rich 
sources 

Provide academic 
rigor in all courses 


To increase 
students’ ability 
to consistently 
locate and use 
syllabi, and 
increase students’ 
understanding of 
course 
components and 
expectations 
through standard 
syllabi 
development 
practices 


To continually 
ensure that 
content is 
sufficient, 
updated, and 
includes current 
and media-rich 
resources 


Offer 
academically 
rigorous 
programs that 
appropriate 
prepare learners 


- Develop and implement 
standard syllabi components - 
Develop and implement 
standard course design practices 
for the location of syllabi within 
the course 


Develop regular review and 
updating processes and 
schedules for course content 
and materials 


- Review activities and 
assignments in courses to ensure 
they provide for higher levels of 
thinking (Bloom’s Taxonomy) 

- Add activities and assignments 


es 


- Standard syllabi 
components are 
developed and 
implemented - 
Standard course 
design practices for 
the location of 
syllabi within the 
course are 
developed and 
implemented 


Course content and 
materials are 
regularly reviewed 
for currency and 
resource content 
and updated as 
appropriate 


Course content 
meets appropriate 
level of academic 
rigor 


- Quality course 
reviews indicate 
that standard 
syllabi 
components are 
used and that 
syllabi are placed 
appropriately 
within the course 
design - Student 
feedback 
indicates that 
syllabi 
components are 
present and 
identifiable, and 
that syllabi are 
consistently 
located within 
their courses 
Reviews of course 
content and 
materials are 
completed and 
resources are 
updated with 
current and 
media-rich 
content 
Activities and 
assignment 
included in 
courses offer 
higher order 
thinking 


Faculty, 
Students 


Faculty 


Faculty 
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higher levels of 
thinking 


Include student 
feedback in 
innovative 
pedagogical and 
technological 


Innovation 


practices 


Communicatio | Identify and define 

n a list of core 
definitions as they 
apply to the 


institution 


Provide 
opportunities for 
students to give 
feedback on 
implemented 
practices, and for 
them to share 
their thoughts, 
needs, and ideas 
on innovation 


Consider the 
components and 
measurements 
associated with 
terms used for 
various iterations 
of online learning, 
identify, and 
define terms that 
will be 
consistently used 


To increase 
student 
awareness of 
innovative 
practices and 
incorporate their 
feedback as 
appropriate in 
continual 
improvement of 
curriculum design 
and planning 
innovation 

To increase the 
clarity and 
effectiveness of 
communication 
regarding the 
various iterations 
and components 
associated with 
online learning 


that provide for higher order 
thinking 
Develop and implement 
opportunities for student 
feedback on the success and 
development of innovative 
practices in curriculum design 
and planning 


- Determine terminology needs 
for each iteration or component 
associated with online learning - 
Define metrics (as appropriate 
and/or needed) and definitions 
of use for each term - 
Communicate terms and 
definitions, and implement 
consistent use of selected 


terminology across all 
stakeholder groups 


Student feedback 
opportunities are 
provided 


- Terminology is 
selected and 
defined - Metrics 
associated with 
terminology (as 
appropriate) are 
selected and 
defined - Selected 
terms and 
definitions are 
communicated to 
stakeholders - 
Consistent use of 
selected terms is 
implemented 


Student feedback 
is evaluated and 


Students 


incorporated into 
decision-making 
on innovative 
practices 


Selected 
terminology is 


Administrators/ 
Staff, Faculty 
consistently used 

across 

stakeholder 

groups and 

accurately used 

for various online 

learning instances 


Engagement 


Expectation 
Setting 


Increase 
opportunities for 
students to interact 
with their peers 
and faculty, 
including 
academically 
challenging 
components 


Expectations for 
student activity are 
clearly defined and 
communicated to 
students 


Expectations for 
responses from the 
faculty are clearly 
defined and shared 


with students 


Implement both 
formal and 
informal 
opportunities for 
students to 
interact with each 
other and faculty, 
and to engage 
with academically 
challenging 
interactivity 


Develop, define, 
and communicate 
expectations for 
student activity 


Establish 
expectations for 
responsiveness 


To increase 
student 
engagement with 
course content, 
their peers, and 
faculty, including 
academically 
challenging 
components 


To increase 
presence, 
communication, 
and awareness of 
student activity 
expectations to 
support 
successful 
student learning 


To set 
expectations 
regarding when 
students can 
expect responses 
from the teacher 


Include intentional opportunities 
for academically challenging 
interactions with students and 
faculty in course design and 
delivery 


- Develop and define student 
activity expectations - 
Communicate expectations to 
students 


Develop, define and 
communicate expectations for 
when students can expect 
responses (to emails, phone 
calls, etc.) from the teacher 


Online courses 
provide interactive 
opportunities for 
students and the 
peers as well as 
faculty, and these 
opportunities 
include 
academically 
challenging 
components 


- Student activity 
expectations are 
present - Student 
activity 
expectations are 
communicated to 
students 


Established 
parameters for 
when teachers will 
respond to student 
communications 


- Quality course 
reviews indicate 
the presence of 
academically 
challenging 
interactive 
opportunities - 
Student 
evaluations and 
feedback indicate 
that interactive 
opportunities are 
present and 
academically 
challenging 

- Student activity 
expectations are 
evident in quality 
course review - 
Student 
evaluations and 
feedback indicate 
that expectations 
are present an 
effectively 
communicated 
Teachers 
responding to 
student 
communications 
within set 
parameters 


Faculty, 
Students 


Faculty 


Faculty 
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Outcomes 


Measure, review, 
and communicate 
student learning 
outcomes in online 
courses, and 
provide 
comparative onsite 
data as 
appropriate 


Rubrics for all 
students learning 
activities and 
assignments 


- Develop and 
implement 
student learning 
outcome metrics 
for online courses 
- Communicate 
findings, 
including onsite 
comparisons as 
appropriate 


Clearly articulate 
criteria for 
assignments 


To develop and 
communicate an 
understanding of 
student learning 
outcomes in 
online courses, 
including 
comparative 
onsite data as 
appropriate, to 
aid in both 
continual course 
improvements as 
well as to provide 
accurate 
perceptions of the 
efficacy of online 
courses 


To provide 
students with 
clearly defined 
criteria for the 
grading of all 
assignments 


- Develop student learning 
outcome metrics for online 
courses - Measure student 
learning outcomes in online 


courses - Communicate findings, 


including comparative onsite 
data as appropriate - Use 
findings to aid in continual 
improvement of courses 


Develop rubrics linked to 
student learning outcomes to 
show grading criteria 


- Student learning 
outcome metrics 
are developed and 
implemented for 
online courses - 
Results of these 
metrics and 
comparative data 
are communicated 
- Findings are 
utilized in 
continual 
improvement 
processes 


Increased 
awareness of 
grading criteria 


- Student learning 
outcomes for 
online courses are 
measured - 
Results are 
communicated 
and include 
relevant 
comparisons to 
onsite courses - 
Findings result in 
continual 
improvement for 
online courses - 
Administrators/st 
aff and faculty 
indicate that they 
understand 
student learning 
outcomes data for 
online courses 
and their 
perceptions 
regarding the 
efficacy of online 
courses are in 
alignment with 
findings 

Rubrics Faculty 
developed and 

provided for all 

assignments and 

activities 


Administrators/ 
Staff, Faculty 


Course 
Interaction 


Feedback 


Opportunities for 
the student to get 
to know the 
teacher 


Feedback provision 
and methods, 
timing, and 
connection to 
learning objectives 
is present and 
communicated to 
students 


Establish online 
presence of the 
educator 


Develop and 
communicate 
methods for 
providing 
feedback, and set 
expectations for 
timing and 


explanations of 


how feedback is 
related to 
learning 
objectives 


To provide 
opportunities for 
the student to 
better know their 
teacher 


To support 
successful 
student learning 
outcomes by 
establishing and 
communicating 
clear processes 
and expectation 
for feedback 


Include an instructor biography 
and contact information 

Share expertise with students 
through bringing outside 
resources (especially those 
written/prepared by the 
educator) 

Regularly participate in 
discussions or other 
collaborative activities 

- Develop processes and 
methods of feedback provision 
in alignment with course 
activities - Communicate 
feedback processes and 
methods, and set expectations 
including timing and how 
feedback is connected to 
learning objectives, to students 


Increased 
engagement 
between the 
instructor and 
students 


- Feedback 
processes and 
methods are 
developed and 
implemented - 
Communication to 
students regarding 
feedback 
processes, 
methods, 
connection to 
learning objectives, 
and expectations is 
provided 


Instructor 
biography and 
contact 
information 
shared 

Instructor regular 
contributing in 
the virtual 
classroom 


- Processes and 
methods of 
feedback are 
present and 
implemented - 
Communication 
to students 
regarding 
feedback is 
present and 
implemented - 
Student 
evaluations 
indicate that 
feedback 
processes, 
methods, and 
provision are 
implemented, 
understood, and 
support learning 


Faculty 


Faculty 
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Measure student 
experiences and 
quality perceptions 
as an integrated 
component of 
course design, and 
use the resulting 
data in continual 
improvement 
processes 


Resources An increased 
understanding of 
student 
perceptions and 
feedback regarding 


course resources 


Develop and 
implement 
measures of 
student 
experience and 
quality 
perceptions 


- Develop and 
implement 
evaluation 
procedures to 
gather student 
perceptions of 
course resources - 
Include this data 
in continual 
course 
improvement 


To gain an 


understanding of 


student 
experiences and 
quality 
perceptions that 
are used in 
continual course 
improvement 
processes 


To understand 
student 
perceptions 
regarding course 
resources for the 
purpose of 
improving 
courses to better 
meet student 
learning needs 


Develop and implement 
measures of student experience 
and quality perceptions 


- Develop and implement 
evaluation procedures regarding 
student perceptions of course 
resources - Utilize resulting data 
to inform continual course 
improvements 


- Measures of 
student experience 
and quality 
perceptions are 
present and 
utilized - Resulting 
data is used in the 
continual 
improvement of 
courses 


- Evaluation 
procedures are 
developed and 
implemented - 
Evaluation results 
are utilized for 
continual course 
improvements 


- Measures of 
student 
experience and 
quality 
perceptions 
enhance 
understanding 
among 
administrators/st 
aff and faculty 
regarding these 
items - Student 
experience and 
quality 
perception 
feedback results 
in course 
improvements 
addressing 
related course 
components 

- Evaluation 
processes result 
in student 
feedback on 
perceptions of 
course resources - 
Results are 
integrated into 
course 
improvement 
processes, which 
reflect inclusion 


Faculty 


Administrators/ 
Staff, Faculty 


Innovation 


Assessment 
Strategies 


Provide students 
with information 
about innovations 
in online teaching 
and learning, and 
gather student 
feedback regarding 
the innovative 
practices 


Clearly identify 
assessments and 
identify their 
purposes and 
learning outcomes 
to students 


processes 


Provide 
communications 
and feedback 
opportunities to 
students 
regarding 
innovative online 
teaching and 
learning practices 


Develop and 
implement 
course-level 
communications 
to students 
regarding 
assessments, 
their purposes, 
and the learning 
outcomes 
addressed by 
each 


To increase 
student 
awareness of 
innovative 
practices and 
incorporate their 
feedback as 
appropriate in 
continual 
improvement and 
innovation 


To increase 
student 
understanding of 
assessment 
practices in order 
to support 
successful 


student learning 


outcomes 


- Increase student awareness of 
innovation in online teaching 
and learning - Develop and 
implement opportunities for 
student feedback on the success 
and development of innovative 


practices 


Faculty provide communications 
to students via various methods 


(syllabi, email, LMS 


communications, etc.) 
identifying assessments, their 
purposes, and the learning 


objectives for each 


- Communications 
regarding 
innovation are 
provided to 
students - Student 
feedback 
opportunities are 
provided 


Faculty provide 
communications to 
students regarding 
assessments 


of appropriate 
modifications 
based on 
identified needs 


- Student 
feedback reflects 
an increased 
awareness of 
innovation in 
online teaching 
and learning - 
Student feedback 
is evaluated and 
incorporated into 
decision-making 
on innovative 
practices 
Communications 
are provided to 
students 
regarding 
assessments - 
Student 
evaluations and 
feedback indicate 
increased 
awareness and 
understanding of 
assessments, 
their purposes, 
and the 
associated 
learning 
outcomes 


Students 


Faculty, 
Students 
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Assessment 
Processes 


Ensure alignment 
of learning 
objectives and 
assessments 


- Implement course 
design best 
practices regarding 
assessment, 
including delivery 
and placement - 
Align 
communications 
and materials, 
including criteria, 
assignments, 
expectations, and 
grading rubrics 
with best practices 


Ensure alignment 
of learning 
objectives and 
assessments 
through quality 
course review 
processes 


Ensure that 
assessments and 
related 
communications 
in online courses 
follow best 
practices 


To support 
successful 
student learning 
through 
alignment of 
learning 
objectives and 
assessments 

To support 
successful 
student learning 
outcomes 
through the use of 
effective course 
design and 
communication 
practices related 
to assessment 


Include alignment of learning 
objectives and assessments in 
quality course reviews 


- Align course assessments with 
effective course design, 
including delivery and 
placement - Ensure that criteria, 
assignments, expectations, and 
grading rubrics are present and 
that these elements are clearly 
communicated to students 


Learning objectives 
and assessments 
are aligned 


- Course 
assessment 
delivery and 
placement follow 
course design 
standards - 
Criteria, 
assignments, 
expectations, and 
grading rubrics are 
present and 
communicated to 
students 


Successful Faculty 
student outcomes 
increase asa 
result of 
effectively aligned 
learning 
objectives and 
assessments 

- Effective delivery | Faculty 
and placement of 
assessments is 

evident in quality 

course reviews - 
Communications 

regarding criteria, 
assignments, 

expectations, and 

grading rubrics 

are provided to 

students - 

Student 

evaluations and 

feedback reflect 

presence and 

understanding of 

criteria, 

assignments, 

expectations, and 

grading rubrics 


Assessment 
Methods 


Feedback provision 


and methods, 
timing, and 
connection to 
assessments is 
present and 
communicated to 
students 


Provide 
professional 
development 
opportunities on 
the design and 
delivery of peer- 
and self- 
assessment 
methods for 
students 


Develop and 
communicate 
methods for 
providing 
feedback, and set 
expectations for 
timing and 
explanations of 
how feedback is 
connected to 
each assessment 


Develop and 
deliver 
professional 
development 
opportunities for 
faculty on the 
design and 
delivery of peer- 
and self- 
assessment 


To support 
successful 
student learning 
outcomes by 
establishing and 
communicating 
clear processes 
and expectation 
for feedback 


To increase the 
effective use of 
peer- and self- 
assessments in 
courses for the 
purpose of 
supporting 
student learning 


- Develop processes and 
methods of feedback provision 
for each assessment - 
Communicate feedback 
processes and methods, and set 
expectations including timing 
and how feedback is connected 
to assessments, to students 


Develop and deliver professional 
development opportunities for 
faculty on the design and 
delivery of peer- and self- 
assessment methods for 
students 


EE 


- Feedback 
processes and 
methods are 
developed and 
implemented - 
Communication to 
students regarding 
feedback 
processes, 
methods, 
connection to 
assessment, and 
expectations is 
provided 


Professional 
development 
opportunities are 
developed and 
provided to faculty 
on peer- and self- 
assessment 
methods for 
students 


- Processes and 
methods of 
feedback for 
assessments are 
present and 
implemented - 
Communication 
to students 
regarding 
feedback is 
present and 
implemented - 
Student 
evaluations 
indicate that 
feedback 
processes, 
methods, and 
provision are 
implemented, 
understood, and 
support learning 
- Faculty 
participate in 
professional 
development 
opportunities - 
Faculty develop 
and implement 
peer- and/or self- 
assessment 


Faculty 


Faculty 
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Assessment 
Types 


Innovation 


Assessments for 
higher order 
thinking 


- Encourage the 
discovery, 
dissemination, and 
implementation of 
innovative 
assessment 
strategies, 
processes, 
methods, and 
types - Provide 
professional 
development on 
the design, 
development, and 
delivery of 
innovative 
assessment 


methods for 
students 


Ensure 
knowledge 
acquisition 


- Explicitly 
encourage 
innovative 
assessment 
through 
communications, 
activities, and 
professional 
development - 
Develop and 
implement 
professional 
development 
opportunities on 
innovative 
assessment 


Implement 
assessments to 


reflect higher 


order thinking 


To increase the 
design, 
development, and 
delivery of 
innovative 
assessment 


Develop assessments (e.g., 
papers, projects, group 
activities, etc.) that reflect 
knowledge acquisition at a 
higher level 


- Develop and implement 
strategies for the discovery and 
dissemination of innovative 
assessment techniques - 
Develop and implement 
professional development 
opportunities for faculty on the 
design, development, and 
implementation of innovative 
assessment 


Learners acquire 
needed knowledge 


- Innovation 
relevant to 
assessment is 
actively 
encouraged - 
Professional 
development 
opportunities 
related to 
innovative 
assessment are 
provided 


opportunities for 
students 


Courses include Faculty 
assessment 
activities that fit 
higher order 
thinking on 
Bloom’s 
Taxonomy 

- Clear Faculty 
encouragement 

of innovative 

assessment is 
communicated to 

faculty - Faculty 

participate in 

professional 

development 

opportunities 

related to 

assessment - 

Faculty 

implement and 

measure 

innovative 

assessment 

practices in their 

courses 


Technology 


Centralized 
Online 
Education 
Infrastructure 
(SSO, LMS, 
etc.,) 


Internet 
Access/Reliabili 


ty 


Evaluate 
technology 
infrastructure to 
ensure that it is 
sufficient to meet 
demand; create 
and implement 
improvement plans 
as appropriate 


Flexible options for 
completing online 
courses to account 
for internet 
reliability and/or 
student needs 


- Evaluate 
technology 
infrastructure - 
Utilize evaluation 
results to create 
and implement 
any necessary 
improvement 
plans 


- Develop and 
implement 
flexible course 
participation and 
completion 
options - 
Collaborations 
between faculty 
and technology 
administrators/st 
aff to support 
flexible course 
options 


To ensure that 
technology 
infrastructure is 
sufficient to meet 
current and 
anticipated 
demands, 
increasing the 
likelihood of 
smooth online 
learning 
experiences for 
faculty and 
students 


To improve the 
ability of faculty 
and students to 
successfully 
engage in online 
education 


- Evaluate technology 
infrastructure - Utilize evaluation 
results to create and implement 
improvement plans as needed 


Develop, implement, and 
support flexible options for 
course participation and 
completion 


- Technology 
infrastructure is 
evaluated for 
ability to meet 
demand - Results 
are utilized to 
create and 
implement any 
necessary plans for 
improvement 


Presence and 
awareness of 
flexible course 
participation and 
completion options 


- Technology 
capabilities and 
capacities are 
understood and 
needs identified - 
Technology 
infrastructure is 
improved as 
needed to 
support demand - 
Technology 
infrastructure 
adequately meets 
the needs of 
faculty and 
students 

- Faculty and 
student use of 
flexible options to 
support online 
learning - 
Evaluations and 
feedback indicate 
efficacy and result 
in 
recommendation 
s for potential 
improvements 


Administrators, 
Staff 


Administrators, 
Staff, Faculty 
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ITSM 
Compliance 


Modality 


Faculty 
development and 
support materials 
on the academic 
technologies 
supported by IT 


Understand 
student 
experiences, needs, 
and perceptions 
regarding 
synchronous and 
asynchronous 
learning, and 
utilize those 
findings to inform 
course design and 
delivery decisions 


Develop and 
disseminate 
faculty 
development and 
support materials 
related to 
academic 
technologies’ use 
and available 
support through 
IT 


- Evaluate student 
experiences, 
needs, and 
perceptions 
regarding 
synchronous and 
asynchronous 
learning in online 
courses - Include 
findings in course 
design and 
delivery decision- 


To ensure 
awareness and 
support for 
academic 
technologies, 
including 
instructional 
design support 
and just-in-time 
technical 
resources 


To support 
successful 
student learning 
experiences 
through the 
evaluation of 
synchronous and 
asynchronous 
experiences, 
needs, and 
perceptions and 


Develop and disseminate faculty 
development and support 
materials related to academic 
technologies supported by IT 


- Gather student feedback on 
their experiences, needs, and 
perceptions regarding 
synchronous and asynchronous 


online learning - Evaluate 


feedback and incorporate 
findings in decision-making 
processes for online course 
design and delivery 


- Faculty 
development and 
support materials 
related to 
academic 
technologies are 
developed - 
Communications 
are provided 
regarding these 
resources to 
increase faculty 
awareness and use 
of academic 
technologies and 
associated support 
needs 

- Student feedback 
is collected - 
Feedback is 
evaluated for 
student needs - 
Needs are 
addressed through 
course design and 
delivery 


- Faculty 
development and 
support materials 
related to 
academic 
technologies are 
developed, 
disseminated, 
and accessed by 
faculty - Increased 
use of academic 
technologies 
and/or support 
resources 


- Student 
feedback 
regarding 
experiences, 
needs, and 
perceptions of 
synchronous and 
asynchronous 
online learning is 
regularly 
collected, 
evaluated, and 
incorporated into 


Faculty 
Students 


Operability/ 
Security 


Coverage 


Technology 
Plan 


Regular monitoring 
of systems 


Regular monitoring 


Increase awareness 
of technology plans 
and how they 
connect with and 
support online 
education 


making 


Operable and 
secure technology 
to support online 
learning 


Continuous 
coverage and 
access to online 
learning 
technology 


Develop and 
implementa 
communication 
plan for 
administrators/st 
aff and faculty 
regarding 
technology plans 
and online 
education 


implementation 


of effective course 


design and 
delivery to 
support student 
needs 


To ensure the 
regular 
functioning and 
security of 
technology to 
support online 
learning 


To ensure that 
administrators, 
staff, faculty, and 
students are able 
to access 
technology 
needed for online 
courses 

To increase 
awareness and 
understanding of 
technology plans, 
particularly as 
related to online 
education in 
order to better 
support 


Establish protocols for regular 
review of the learning 
management system and other 
technologies to ensure they 
continue to operate as required 
and remain secure 


Establish programs for review of 
technologies to make sure 
access is maintained 


Develop and implement a 
communication plan for 
administrators/staff and faculty 
regarding technology plans and 
online education 


Functioning 


technology to 


support online 
learning 


Technologies 
available to 
administrators, 
staff, faculty, and 
students 


Communications 
are provided to 
administrators/staf 
f and faculty 
regarding 
technology plans 
and online 
education 


course design and 
delivery - Student 
feedback and 
course outcomes 
indicate effective 
course design and 
delivery 


- No, or limited, 
downtime for the 


Administrators, 
Staff 

learning 

management 

system or other 

technologies 

- No security 

breaches with the 

learning 

management 

system or other 

technologies 

Administrators, 
Staff, Faculty, 
Students 


Coverage and 
access to online 
learning 
technologies 
maintained 


Administrators/st 
aff and faculty 
report increased 


Administrators, 
Staff, Faculty 


awareness of 
technology plans 
and how they 
connect with and 
support online 
education 
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Innovation 


Student 
Orientation & 
Support 


Include student 
feedback regarding 
the use and 
implementation of 
innovative 
technology 


- Reduce response 
time to address 
student 
problems/complai 
nts - Increased 
availability of 
students support 
services to address 
the specific needs 
of students in 


online learning 


environments 


Provide 
opportunities for 
students to give 
feedback on 
implemented 
tools/practices, 
and for them to 
share their 
thoughts, needs, 
and ideas on 
innovation 


Ensure 
technology 
training, 
tools/hardware, 
and resources to 
support best 
practices and 
academic success 
are available to all 
faculty and 
students engaged 
in online learning 


stakeholder 


needs 


To increase 
student 
awareness of 
innovative 
practices and 
incorporate their 
feedback as 
appropriate in 
continual 
improvement of 
technology 
innovation 

To enrich and 
support the 
student and 
faculty online 
learning 
experience to 
ensure successful 
student outcomes 


Develop and implement 
opportunities for student 
feedback on the success and 
development of innovative 
practices in technology 


- Extend the breadth and depth 
of orientation topics - Introduce 
Online Learning Readiness 
assessment prior to enrollment 
in online courses - Provide 
voluntary, interactive training 
opportunities - Invest in 
technology/staffing: to scale, 
streamline student 
problem/complaint mechanisms 
and reduce response time; to 
deliver online resources 
comparable to those available to 
on-campus students; and to 
ensure all faculty and students in 


Student feedback 
opportunities are 
provided 


- The technology 
and teaching and 
learning 
requirements for 
online courses is 
clearly 
communicated to 
faculty and 
students - The 
availability of 
student support 
resources is clearly 
communicated to 
faculty, advisors, 
and students 


Student feedback 
is evaluated and 
incorporated into 
decision-making 
on innovative 
practices 


- Students’ 
surveys and 
feedback report 
prompt responses 
to 
problems/compla 
ints - Students’ 
surveys and 
feedback report 
the availability 
and helpfulness 
of support 
services 


Students 


Faculty, 
Students, 
Advisors, 
Information 
Technology 
Staff, 
Instructional 
Designers 


Student support 
offered to online 
students 


Equity - Increase faculty 
and students’ 
knowledge of 
technologies and 
online learning 
best practices - 
Increase training 
for faculty to 
increase 
engagement from 
all students - 
Instructional 
designers to assist 
faculty in 
developing courses 
and using 
tools/apps that 
increase 
engagement and 
limit distractions - 
Increase 
technology, 
including assistive 


Provide a seamless 
process for faculty 
and students to 


a 


technology access - 


Provide student 
services to online 
learners that is 
equivalent to 
services provided 
in-person 


Ensure 
technology 
training, 
tools/hardware, 
and resources to 
support best 
practices and 
academic success 


are available to all 


faculty and 
students engaged 
in online learning 


To provide 
services through 
virtual modality 
for students 
unable to access 
in-person 


To enrich and 
support the 
student and 
faculty online 
learning 
experience to 
ensure successful 
student outcomes 


online courses have required 
technology tools 


Review existing student services 


and provide options for online 
students to access 


- Extend the breadth and depth 
of orientation topics - Introduce 
Online Learning Readiness 
assessment prior to enrollment 
in online courses - Provide 
training opportunities using 
assistive technologies - Ensure 
instructional design support 
provides faculty the tools and 
best practices that support 
engagement from diverse 
student groups - Invest in 
technology and training to 
ensure all faculty and students 


have access to accommodations 


or assistive technologies to 
deliver an equitable learning 
environment including the 
course content and associated 
resources 


Improved service 
to online students 


Ensure early 
communication of 
the availability of 
instructional 
design support, 
assistive 
technology, other 
support services to 
students and 
faculty 


Online students 
able to access 
needed services 
through virtual 
modality 


Students’ surveys 
and feedback 
report that the 
necessary tools, 
training, 
resources, and 
assistance was 
available for them 
to participate in 
the online 
learning 
environment 


Administrators, 
staff 


Faculty, 
Students, 
Advisors, 
Information 
Technology 
Staff, 
Instructional 
Designers 


161 


0) (je: ONLINE LEARNING™ 
CONSORTIUM 


Accessibility 


Compliance 
Standards 


162 


request 
accommodation or 
other support to 
create equitable 
learning 
opportunities 


- Investigate 
student and faculty 
needs - Monitor 
legal requirements 
- Explore assistive 
and adaptive 
technology - 
Establish policies 
and practices to 
support course 
designers and 
provide 
accommodations 
for students and 
faculty with a 
disability - Ensure 
accessibility 
features and 
functions are built 
in from the 
beginning, rather 
than approached 
retroactively 

- Include all 
stakeholders in the 
implementation of 


To ensure online 
learning practices 
adhere to 
accessibility laws, 
policies, and 
standards 


Verify that all 


university 
personnel and 


To provide an 
equitable learning 
environment for 
students and 
faculty that 
follows best 
practices for 
digital learning 


To provide a 
secure, 
trustworthy 


Form an Accessibility Task Force 
composed of IT staff, Faculty, 
college Administrators/Staff to 
investigate current technology 
and capacity for accessibility - 
Provide recommendations to the 
appropriate university 
administrators and the Ministry 
of Education - Make use of UDL 


- Establish annual review of the 
institution’s compliance goals - 
Implement compliance training 


Laws, policies, and 
practices are in 
place to ensure 
that the technology 
and resources for 
online courses is 
available to faculty 
and students 


Implementation of 
training and annual 


The Accessibility 
Task Force 
surveys the 
current online 
learning 
environment and 
submits its 
findings and 
recommendation 
sto the 
appropriate 
administrators, 
departments, and 
divisions of 
government 


- Successful 
completion of 
compliance 


Ministry of 
Education, 
Administrators, 
Staff, 
Information 
Technology 
Staff, 
Instructional 
Designers, 
Faculty 


Administrators, 
Staff, 
Information 


Innovation 


compliance 
standards - Provide 
training about 
compliance 
standards and 
goals - 
Communicate 
compliance goals 
to the university 
community 


- Create capacity 
and scale for online 
learning - Make use 
of third-party 
vendors as 
appropriate and 
cost effective - 
Improve the online 
learning 
experience for 
students - 
Investigate the 
delivery of health 
and well-being 
check-ins, 
automated 
feedback, and 


technology training 


through vendors - 
Make use of shared 
tools through 
strategic approach 
to learning tools 
adoption 


staff are aware of 
and maintaining 
compliance goals 


Develop policies 
and practices to 
investigate 
innovative tools 
and implement an 
adoption strategy 
for tools and 
practices 


learning 
environment for 
the university 
community 


To provide a 
secure, engaging 
learning 
experience that 
leads to positive 


student learning 


outcomes 


for administrators/staff, faculty, 
and students 


- Create a Learning Innovation 
Taskforce (or new unit) to 
investigate tools and pedagogy - 
Develop policies and practices 
that deliver cost-effective, 
innovative tools that provide 
capacity and scale for online 
learning 


review of 
compliance goals 


Increased capacity 
for online learning 
and delivery of 
student support 
resources 


training - Effective 
communication 
of compliance 
goals - 
Completion of 
annual review of 
compliance 
standards and 
goals 


Students’ surveys 
and feedback 
report positive 
interactions with 
course tools and 
provide 
opportunities for 
timely, engaging 
faculty and peer 
interactions 


Technology 
Staff, Faculty, 
Educational 
Technology 
Vendors 


Administrators, 
Staff, Faculty, 
Students, 
Information 
Technology 
Staff, 
Instructional 
Designers, 
Educational 
Technology 
Vendors 
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Technical 
Assistance 


Improve 
communication 
and technical 
assistance 
mechanisms for 
students 


Improve 
communication 
and technical 
assistance 
mechanisms for 
faculty 


Create a 
centralized 
support space 
through a 
website; 
Resources 
explicitly shared 
and stressed by 
faculty in syllabi 
and online 
course, and the 
institution 


Create a 
centralized 
support space 
through a 
website; 
Resources 
explicitly shared 
by staff 


To remove 
barriers for course 
progress and 
develop resources 
for common 
issues 


To remove 
barriers for 
faculty teaching 
in an online 
modality 


- Create an advisory team of 
stakeholders across campus 
including student members to 
guide development of needed 
resources - Create a centralized 
website with curated resources 
(e.g., guides, videos, frequently 
asked questions) and support 
mechanisms (ex: live chat, 
chatbot) - Communicate 
resources through institutional 
channels, online courses, and 
syllabi 


- Create an advisory team of 
stakeholders across campus 
including faculty to guide 
development of needed 
resources 

- Create a centralized website 
with curated resources (e.g., 
guides, videos, frequently asked 
questions) and support 
mechanisms (ex: live chat, 


- Stakeholder 
expertise and 
listening to student 
needs - Centralized 
website available 
with curated 
resources and 
support 
mechanisms, and 
updated to ensure 
current - Resources 
shared through 
institutional 
channels, online 
courses, and syllabi 
Centralized 
website available 
with curated 
resources and 
support 
mechanisms, and 
updated to ensure 
current 


- Website piloted 
and shared for 
feedback - Future 
student technical 
issues curated to 
update website - 
Each syllabus and 
online course has 
common support 
language - 
Institution shares 
resources through 
a variety of 
channels 


- Website created 
- Faculty technical 
issues curated to 
update website 


Administrators, 
Staff, Faculty, 
Students 


Administrators, 
staff, faculty 


Professional 
Development 


Improved 
communication 
between 
stakeholder groups 


Increase 
opportunity for 
faculty to complete 
professional 
development and 
remove barriers for 
support resources 


Create 
mechanisms for a 
feedback loop 


Transition 
synchronous 
training to be 
more 
asynchronous 
and curate 
resources ina 
centralized space 


To improve 
communication 
and better 
understand 
where pain points 
exist and how to 
better address 
faculty and 
student needs to 
improve overall 
course outcomes 


To provide 
greater flexibility 
and more options 
to complete 
professional 
development and 
remove barriers 
for support 
resources 


chatbot) 

- Communicate resources 
through institutional channels 

- Create opportunities for 
stakeholder feedback (e.g., post- 
event surveys, focus groups, 
town hall meetings) - Conduct a 
needs assessment to help 
illuminate areas of where faculty 
need more support, resources, 
etc. - Create a mechanism for 
students to share their 
perspectives (e.g., town hall 
meetings, focus groups, survey) 


- Break longer synchronous 
training sessions into more 
manageable parts that can be 
completed over a longer period 
of time - Create short courses 
using the resident learning 
management system (LMS) (ex: 
Blackboard) around specific 
training topics to create a space 
for engagement, practice with 
tools and features, and share 
ideas, ask questions, and engage 
with the content - Curate, post, 
and communicate resources 
pertaining to moving teaching 
online in one space (such asa 
website for a Center for Teaching 
and Learning) 


- Broad faculty 
participation in 
needs assessment - 
Timely creation of 
feedback loops 
that are 
maintained and 
communicated 
each semester - 
Feedback 
implemented 


- Opportunities for 
faculty to ask 
questions ina 
synchronous 
format (e.g., 
weekly live session 
or virtual office 
hours) - Modular 
content that can be 
completed in 
shorter chunks of 
time - Technical 
support structure 
for short courses - 
Advisory group of 
faculty and staff to 
provide feedback 
on curated 
resources - Faculty 
are made aware of 
resources through 
a variety of 
channels (e.g., 
departments, 
emails, etc.) 


Administrators, 
Staff, Faculty 


- Response rate of 
50% or greater - 
Improved faculty 
satisfaction with 
professional 
development and 
student 
satisfaction with 
courses 


Administrators, 
Staff 


- Increased faculty 
participation in 
professional 
development - 
Improved student 
outcomes 
(depending on 
topic) 
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Regular access to 
professional 
development 


Orientation Improve access to 
orientations for 
widespread/comm 
on technologies 

Mentoring Preserve and 


improve 
opportunities for 
peer mentoring 


Faculty training 
on course design 
and teaching 
online 


Develop and 
share common 
orientation 
modules for 
institutions 


Develop a virtual 
mentoring 
program 


To provide 
regular access to 
professional 
development 
specific to online 
learning 


To decrease 
resistance and 
challenges among 
faculty for online 
learning 
technologies and 
improve student 
outcomes 


To provide 
opportunities for 
faculty to 
continue to 
receive support 
and guidance 
during remote 
teaching and 
learning, 
including 


- Identify professional 
development needs 

- Provide workshops on specific 
needs (either in-house or 
through provider) 


- Identify widespread 
technologies and determine 
most critical to develop - Partner 
with different institutions to 
develop and pilot orientation 
modules - Publicly host and 
share modules with institutions 


- Communicate with faculty to 
identify mentoring needs - Pair 
mentors with mentees based on 
needs - Provide resources and 
incentives where possible and 
appropriate 


Faculty better 
trained to teach 
online 


- Most critical 
technologies 
identified - 
Periodically 
updated to include 
new technologies - 
Piloted with 
institutions to 
provide feedback - 
Orientations easily 
accessible and 
shared with key 
personnel at 
institutions 

- Needs identified - 
Faculty volunteers 
to be mentors and 
mentees - Means 
for incentivizing 
participation, if 
possible - 
Feedback loop to 
improve program 


Faculty provided 
with access to 


Faculty 


quality training 


- Number of Administrators, 
modules Staff, Ministry of 
developed and Education, 
updated National E- 
periodically - Learning Center 
Number of 


modules accessed 
and implemented 
by institutions 


- Number of Administrators, 


mentors and Staff, Faculty 
mentees - 

Academic output 

(e.g., 

publications, 

grants, etc.) - 

Faculty retention 

- Faculty health 


Innovation 


Greater awareness 
and use of 
innovative 
technologies 


Develop a faculty 
learning 
community 
program 


maintaining 
academic 
momentum 


To build capacity 
around online 


learning tools and 


approaches 


- Bottom-up: faculty/staff self- 
organize into a community (or 
multiple communities) - Top- 
down: institution recruits 
faculty/staff into communities 
and provides centralized way to 
coordinate the groups 


where possible 


- Bottom-up: 
Groups set 
expectations, 
regular meetings, 
participants share 
resources and 
ideas - Top-down: 
Clear expectations 
for program, 
regular meetings, a 
point person to 
coordinate the 
group and ensure 
all needs and 
outcomes met, 
incentives for 
participation 
(stipends, 
resources, 


awards/recognition 


, etc.), recruitment 
from all 
departments to 


and wellbeing 


- Number of Administrators, 


groups - Staff, Faculty 
Participation in 

groups - Diffusion 

of knowledge and 

skills 
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Evaluation of 
Course 
Outcomes and 
Program 
Quality 


Develop and 
implement clear 
processes, 
procedures, and 
policies on course 
and program 
evaluation, 
including regular 
review and 
updating 


Increased 
communication, 
transparency, and 
involvement of 
faculty with regard 
to analytics used 


Processes, 
procedures, and 
policies on course 
and program 
evaluation, 
including review 
and updating, are 
developed and 
implemented 


Communications 
to faculty 
regarding the 
analytics included 
in programmatic 


To maintain 
and/or increase a 
high level of 
adherence to 
processes, 
procedures, and 
policies for course 
and program 
evaluation, and to 
continually 
improve these 
items through 
regular review 
and updating 


To increase 
faculty 
awareness, 
participation, and 
buy-in of 


- Develop and implement course 
and program evaluation 
processes, procedures and 
policies - Regularly review and 
update course and program 
evaluation processes, 
procedures, and policies 


- Communicate with faculty 
regarding analytics used for 
programmatic decision-making - 
Include faculty, when possible, 
in programmatic decision- 


grow champions 


- Course and 
program 
evaluation 
processes, 
procedures, and 
policies are in 
place - Course and 
program 
evaluation 
processes, 
procedures, and 
policies are 
regularly reviewed 
and updated as 
appropriate 


- Communications 
are provided to 


faculty regarding 


programmatic 
decision-making - 
Faculty are 


- Courses and 
programs are 
evaluated in 
alignment with 
established 
processes, 
procedures, and 
policies - Course 
and program 
processes, 
procedures, and 
policies are 
reviewed and 
updated as 
appropriate ona 
regular schedule 
- Faculty indicate 
awareness of 
analytics used for 
programmatic 
decision-making - 
Faculty 


Faculty 


Faculty 


Faculty 
Satisfaction 


for programmatic 
decision-making 


Develop and 
implement 
practices and 
expectations 
regarding faculty 
peer evaluation, 
including 
processes and 
procedures as well 
as professional 
development on 
development, 
implementation, 
and use 
Communications 
to faculty regarding 
evaluation 
opportunities, their 
purpose, and how 
feedback is 
incorporated into 
course and 


decision-making - 


Inclusion of 
faculty in 
programmatic 
analytics and 


decision-making 


when possible 


and appropriate 


Implement 
faculty peer 
evaluation, and 
provide related 
professional 
development 


Create a 
communication 
plan for faculty 
regarding 
evaluation 
opportunities, 
purposes, and 
uses 


programmatic 
decision-making 


To increase the 
use of effective 
peer evaluation 
for faculty in 
order to support 
continued 
evaluation and 
development of 
effective teaching 
and learning 
practices 


To increase 
awareness and 
opportunities for 
faculty 
evaluations 


making 


- Develop and implement 
practices and expectations 
regarding faculty peer 


evaluation - Provide professional 
development on peer evaluation 


to faculty 


Develop and implement a 
communication plan regarding 
evaluation opportunities, 
purposes, and uses for faculty 


FE 


included in 
programmatic 
decision-making 


- Clear practices 
and expectations 
for faculty peer 
evaluation are in 
place - Professional 
development 
opportunities on 
peer evaluation are 
available to faculty 


Communications 
are provided to 
faculty regarding 
evaluation 
opportunities, 
purposes, and uses 


participate in 
programmatic 
decision-making 
processes 


- Faculty peer Faculty 
evaluation is 
effectively 
employed - 
Faculty 
participate in 
professional 
development 
opportunities 
related to peer 
evaluation 


Faculty Faculty 
participate in 
evaluation 


opportunities 
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Staff 
Satisfaction 


Student 
Satisfaction 


Innovation 


program 
improvements 
Communications 
to staff regarding 
evaluation 
opportunities, their 
purpose, and how 
feedback is 
incorporated into 
course and 
program 
improvements 


Communications 
to students 
regarding 
evaluation 
opportunities, their 
purpose, and how 
feedback is 
incorporated into 
course and 
program 
improvements 
Access to research 
and international 
experts 


Create a 
communication 
plan for staff 
regarding 
evaluation 
opportunities, 
purposes, and 
uses 


Create a 
communication 
plan for students 
regarding 
evaluation 
opportunities, 
purposes, and 
uses 


Implement best 
practices 
identified by 
online learning 
experts 


To increase 
awareness and 
opportunities for 
staff evaluations 


To increase 
awareness and 
opportunities for 
student 
evaluations 


To ensure quality 
of online learning 
by benchmarking 
against best 
practices 


Develop and implement a 
communication plan regarding 
evaluation opportunities, 
purposes, and uses for staff 


Develop and implement a 
communication plan regarding 
evaluation opportunities, 
purposes, and uses for students 


-Review quality standards 
(rubrics, scorecards, research, 
etc.) shared by online learning 
experts 

-Apply best practices to online 
learning courses and programs 


Communications 
are provided to 
staff regarding 
evaluation 
opportunities, 
purposes, and uses 


Communications 
are provided to 
students regarding 
evaluation 
opportunities, 
purposes, and uses 


Improved quality of 
online learning 


Staff participate 
in evaluation 
opportunities 


Students 
participate in 
evaluation 
opportunities 


-Use of rubrics or 
scorecards to 
evaluate quality 
of online learning 
-Adoption of best 
practices 


Students 


Administrators, 
staff, faculty 


Appendix V: Recommended Resources 


The following resources (many free) are being provided to support the recommendations provided 


throughout this review. These resources are available through the contributors of this report. 


Dimension 
Leadership 


Resource 


2020 CHLOE Report 
The Changing Landscape of Online Education 
(CHLOE): Navigating the Mainstream 


Contributor 
Quality Matters 


Access and Accessibility in Online Learning: 
Issues in Higher Education and K-12 Contexts 


Online Learning 
Consortium 


Global quality in online, open, flexible and 
technology enhanced education: An analysis of 


strengths, weaknesses, opportunities and 
threats 


International Council for 
Open and Distance 
Education 


Emergency Preparedness and Continuity 
Planning for Administrators 


Online Learning 
Consortium 


APLU COVID-19 Resources 


Association of Public & 
Land-Grant Universities 


Institute for Emerging Leadership in Online 
Learning (IELOL). 


Online Learning 
Consortium 


OLC Leadership in Online Learning Mastery 
Series. 


Online Learning 
Consortium 


Curriculum Design 
& Planning 


Guidance on Open Educational Practices during 


School Closures 


UNESCO IITE 


OLC .OSCQR Course Design Review 


Online Learning 
Consortium 


QM Rubric for Course Review 


Quality Matters 


OLC Instructional Designer Programs - Master 
Series, Certificate, and Courses 


Online Learning 
Consortium 


Online Teaching & 
Learning 


Delivering High Quality Instruction Online in 
Response to COVID-19 


Online Learning 
Consortium 

Every Learner Everywhere 
Association of Public & 
Land-Grant Universities 


Quality Course Teaching & Instructional Practice 


Scorecard. 


Online Learning 
Consortium 


OLC Online Teaching & Advanced Online 
Teaching Certificates 


Online Learning 


_ Consortium 
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OLC New to Online Workshops. 


Online Learning 
Consortium 


QM Teaching Online. Certificate 


Quality Matters 


QM New to Online Teaching Series 


Quality Matters 


Assessment Creating Effective Assessments. workshop Online Learning 
Consortium 

Technology QM Gauging Your Technology Skills workshop Quality Matters 

Student Support OLC Online Student Support scorecard Online Learning 


Consortium 


Training & Support 


OLC Institute for Professional Development. 


Online Learning 
Consortium 


QM Professional Development 


Quality Matters 


Evaluation & 
Continuous 
Improvement 


OLC Quality Scorecard for the Administration of 
Online Learning 


Online Learning 
Consortium 


QM Program Certification 
e Online Program Design 
e Online Teaching Support 
e Online Learner Support 
e Online Learner Success 


Quality Matters 
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Appendix VI: Literature Review 


Literature Review 


Introduction 


Online education has rapidly increased worldwide in the past two decades. Burgeoning demands for 
higher education as well as resource and capacity constraints and the pressures of a global knowledge 
economy have been met with the advent of increasingly more online courses and programs. These 
needs have converged with a period of prolific technology advancements, which have increased 
opportunities for online education and resolved many of the early challenges to teaching and learning 
in an online environment. Enrollments have grown along with availability, and these trends have given 
rise to significant research about effective strategy and pedagogy for online teaching and learning. 


Online in Saudi Arabia 


Beginning with governmental development plans in 1970, education across all sectors has grown 
rapidly, and there are currently 29 public and 56 private universities in Saudi Arabia, as well as hundreds 
of community, women’s, and vocational colleges. Along with the number of higher education 
institutions, demand has continuously grown, and institutions have struggled with a lack of capacity to 
meet total demand (Allahmorad & Zreik, 2020). 


The Saudi Ministry of Higher Education recognized these challenges early on, and has taken clear steps 
to develop solutions, primarily based in online education and technology. Online education has steadily 
increased in Saudi Arabia from the early 2000s on, with a focus on solving two primary challenges: 
bridging the gap between the college-age population/number of high school graduates and the 
capacity of higher education institutions, and providing higher education opportunities to students in 
rural or otherwise underresourced areas (Alturki, 2014; Hamdan, 2014). In 2005, the National eLearning 
Center was established to lead research, development, and implementation of online infrastructure and 
education (NELC, n.d.). In addition, the Ministry of Higher Education has consistently supported the 
exploration and development of online education opportunities through workshops and research, in 
addition to traditional institutions’ efforts to deliver online courses and programs and the development 
of online-only institutions (Hamdan, 2014). 


These efforts (both traditional and online) are reflected in higher education enrollments, which reached 
approximately 1.7 million in 2016, up from fewer than 650,000 in in 2006 (ICEF, 2018). Despite these 
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figures, demand continues to outpace capacity and estimates place the 18-24 year old population 
topping four million by 2025 (currently just over three million) (ICEF, 2018; UNESCO, 2020). 


Terminology 


Before reviewing literature on pertinent topics and issues related to online education, a brief discussion 
of terminology is needed. Worldwide, there are a number of terms used to discuss and describe online 
education, and while they are often used interchangeably, there are some differences. Understanding 
those differences and using the same terms and definitions across institutions and environments can 
help ensure that development, implementation, and measurement of initiatives are as effective as 
possible. Singh and Thurman (2019) found that between 1988 and 2018, 46 different definitions of 18 
common terms were used to describe concepts of online learning, sometimes with contradictory 
implications. Further increasing the need to understand and use standard terms and definitions, Moore, 
Dickson-Deane, and Galyen (2011) found that though terms may be used interchangeably, they often 
include different expectations and perceptions. 


While there is still not broad consensus regarding the selection and use of specific terms in the field, 
there are several key considerations of meaning and perception that can guide those seeking to 
establish standard terms and meaning (Singh & Thurman, 2019). The first and most impactful 
consideration for terms is the use of technology. The wide advent of online capabilities in 1995 marked 
the largest shift in terminology. For example, prior to inclusion of the internet, terms such as distance 
learning were most common, and after, new terms such as e-learning and online learning emerged. 
Rather than replacing former terms, they added to the lexicon, and are still frequently used 
interchangeably. However, the terms convey information and expectations to faculty and students, and 
selecting and consistently using the most appropriate descriptor can be helpful. Additionally, the use 
and expectations of time should be considered. Time plays an important role in the perception of terms, 
as it can indicate synchronicity or asynchronicity, as well as convey expectations regarding meetings 
and interactions (i.e., online, on-campus, blended), and communicate information about the course 
expectations (for example, in a flipped classroom model, expectations are independent learning of 
material but interactive participation). In the continued vein of expectation and perception setting, the 
educational context indicated by various terms should be considered. If the institution offers a variety 
of course contexts, such as for-credit, non-credit, continuing education, and/or professional 
development, using clear and distinct terminology for these differences can help faculty and staff 
develop these different courses appropriately, and can help students select and participate per their 
needs. Finally, interactivity is a key delineator of terminology. The key difference is whether the 
distance interface (internet, video, etc.) is used to provide materials and content, or whether there is a 
virtual learning environment in which all course and learning activities take place. Although there is not 
a broad community consensus about specific terminology across the field, these considerations can 
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assist those seeking to develop common terms across institutions, systems, or larger environments in 
choosing and implementing the terms most appropriate for their contexts. 


Compounding these challenges and accelerating the need for scalable, effective online higher 
education opportunities is the current global Coronavirus pandemic. The pandemic forced educational 
institutions across the world to rapidly shift to remote teaching in the middle of the Spring 2020 term, a 
phenomenon that has now given rise to the need to develop and implement long-term solutions much 
more quickly than many had previously planned. 


Online vs Emergency Remote 
Teaching 


Despite the marked increase in courses delivered through the online modality due to Coronavirus - 
UNESCO estimates that 1.5 billion total students (more than 90% of the student population) were 
shifted to fully online courses - it is important to note that there is a difference between this situation, 
referred to as Emergency Remote Teaching (ERT) and online education (Affouneh, Salha, & Khlaif, 2020; 
Bozkurt & Sharma, 2020; Hodges, Moore, Lockee, Trust, & Bond, 2020; Karalis, 2020; Schlesselman, 
2020). Therefore, it is also necessary to understand the differences and how ERT may translate into 
long-term solutions for scalable online education. 


Hodges, Moore, Lockee, Trust, & Bond (2020) succinctly sum up the primary difference between ERT and 
online learning: “In contrast to experiences that are planned from the beginning and designed to be 
online, emergency remote teaching (ERT) is a temporary shift of instructional delivery to an alternate 
delivery mode due to crisis circumstances.” Schlesselman (2020) elaborates on the meaning of online 
learning, stating that 


“In contrast to emergency remote teaching, online learning is bound by many standards to ensure 
optimal learning. Online learning provides opportunities for community building among students and 
with the instructor; online learning provides opportunities for learner engagement with the material 
through realistic practice, spaced repetition, real-world context, and feedback; online learning provides 
opportunities to support learners. Online learning focuses on the achievement of learning objectives, 
rather than covering content. Online learning also creates a safe environment that addresses the needs 
of all learners, including accessibility.” 


The assumption with ERT is that it is a stopgap designed to provide as much academic continuity as 
possible during an emergent situation, and that courses will return to their previous modalities 
following the crisis. Coronavirus presents a unique challenge for two reasons. First, as Bozkurt and 
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Sharma (2020) and Karalis (2020) point out, this is the first crisis of such scale and severity to occur in 
the current digital knowledge economy. Second, the full impact and trajectory of the coronavirus 
pandemic remains to be seen; educational institutions find themselves in a situation of uncertainty, 
grappling with simultaneous demands to not only provide flexible solutions for the upcoming Fall 2020 
term, but also to design, implement, and scale online education options that were either previously 
unplanned or were not planned with such immediacy (Hodges, Moore, Lockee, Trust, & Bond, 2020; 
Schlesselman, 2020). 


As institutions prepare to make the second large-scale shift resulting from the pandemic, from ERT to 
the design and delivery of online courses, there are several key principles to consider: planning, design, 
and development, pedagogy, quality, communication, technology, student support, and faculty 
support. 


Planning, Design, and Development 


A primary difference between ERT and online learning is the planning and design of the course. For 
online courses, this phase typically ranges from six to nine months or more, and provides a foundation 
for both faculty and students to have a successful learning experience (Hodges, Moore, Lockee, Trust, & 
Bond, 2020; Schlesselman, 2020; Shisley, 2020). This phase is a marked difference in which a course is 
thoughtfully created with the goal of effective online learning as opposed to the hurried translation of 
large quantities of material originally prepared for on-campus instruction and improvised solutions that 
accommodate the unexpected change in modality. 


Effective online course design begins with a focus on learning; alignment between learning objectives, 
instructional activities, and assessments, as well as planned immersive, adaptive, and other innovative 
learning experiences. Studies have shown that misalignment among these items is the primary cause of 
poor outcomes. In an emergent situation, attending to these elements is key: reviewing learning 
objectives and revising as appropriate, communicating changes, and providing aligned materials and 
assessments is the most important step toward ensuring learning continuity (O’Keefe, Rafferty, Gunder, 
& Vignare, 2020). 


It is also critical to consider accessibility when designing an online course, including both web 
accessibility and the broader principles included in Universal Design for Learning (UDL), which shifts the 
focus from accommodations to accessible design for all (CAST, n.d.; O'Keefe, Rafferty, Gunder, & 
Vignare, 2020). UDL is beneficial for both planned online learning as well as ERT (whenever possible). 
For example, providing captioned videos ensures accessibility for deaf or hard-of-hearing students, but 
also provides solutions for students who may be non-native speakers, or working in a loud or 
distracting environment. Lastly, faculty should attend to providing not just an accessible environment, 
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but an equitable one. In course design, equity may be addressed through review for, and replacement 
of, biased language, images, videos, or other materials. Additionally, online course design should 
include explicit attention to equitable design, including academic, pedagogical, psychological, social, 
and technological considerations (Kelly, 2019), and these elements are included in their respective 
sections below. 


Pedagogy 


While the elements of effective pedagogy are similar in on-campus and online environments, the 
delivery, method, and implementation of instructional strategies is markedly different. At a basic level, 
effective pedagogy builds on course design and aligned objectives, activities, and assessments to 
provide regular and high-quality interaction between students and content, students and peers, and 
students and instructors. In addition to interaction, the Community of Inquiry Framework identifies the 
need for instructors to provide social presence (participant identification with the community), 
teaching presence (design and facilitation of meaningful and educationally valuable learning 
outcomes), and cognitive presence (the ability to construct and confirm meaning) (COl, n.d.). 


In an online course, these elements are carefully designed to align with the learning objectives and 
facilitate mastery of the material. However, this type of immersive experience requires planning and 
preparation, and in an ERT environment, it is most useful to focus on translating on-campus strategies 
that lend themselves well to online alternatives, such as group discussions, writing activities, and 
project-based learning (O’Keefe, Rafferty, Gunder, & Vignare, 2020). 


When considering accessibility and pedagogy, UDL is once again a primary tool in ensuring that optimal 
standards are met. By designing learning for all, it is not just the online experience, but interactions that 
impact students. For example, by providing learning opportunities through multiple means 
(slides/notes, video, a variety of activities), all students may access and learn the same material as 
appropriate for their needs (CAST, n.d.; Kelly, 2019). In addition to following best practices for 
interaction and presence, equity in pedagogy may be developed through the intentional creation of an 
inclusive environment, including authentic and equity-focused strategies for teaching and interaction. 


Quality 

Ensuring and maintaining course quality is critical to continued effectiveness in online design, teaching, 
and learning. Course design is foundational for a high-quality course, and faculty may not have the 
knowledge or skills in pedagogy and instructional design to translate their subject-matter expertise 
and/or classroom experience into quality online learning (McGahan, Jackson, & Premer, 2015). There 
are many models which address this potential gap and provide options for evaluation and continuous 
improvement, including the Online Learning Consortium (OLC) Quality Scorecard Suite, Quality Matters 
(QM), and Quality Learning and Teaching (QLT), among others (McGahan, Jackson, & Premer, 2015; 
O’Keefe, Rafferty, Gunder, & Vignare, 2020). 
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Though each approach has variations, the goal is to both ensure that key quality indicators are present 
in the course, and to build confidence in design and pedagogy processes, and there are clear common 
elements among quality measures. These are typically comprised of a combination of course navigation 
and information, course design, accessibility, technology/tools, content, activities, and assessments, 
interaction, and feedback (McGahan, Jackson, & Premer, 2015; OLC, n.d.). 


When designing an online course, these elements may be planned for in advance, and quality 
evaluation and assurance provided. It should be noted that even courses designed for online learning 
should be assessed regularly and continually improved to ensure best practices for quality. In an ERT 
situation, it is helpful for faculty to be familiar with quality standards for online learning, but the focus 
during these events is often just to move forward. Following the initial point of crisis, it may be possible 
but is not always the best solution to convert an ERT course into an online course, and it may be more 
efficient and effective to redesign the course with online quality standards in mind (O’Keefe, Rafferty, 
Gunder, & Vignare, 2020). 


Communication 


Communication, and planning for communication, is important in any course, and is critical in online 
courses, where students and instructors do not necessarily interact synchronously on a regular basis. 
Research has clearly linked high-quality communication between both students and instructors as well 
as students and their peers to higher levels of course engagement, positive outcomes, and student 
satisfaction (Jaggars & Xu, 2016; Leh & Burk, 2016). 


Effective communication in online courses is marked by opportunities for students to connect both 
individually and in groups, and with each other and their instructor(s) in an open, inclusive, and 
equitable environment. In online course design, these opportunities may range from welcome 
messages and icebreaker activities to synchronous and asynchronous communication through various 
media (i.e., video, discussion boards, etc.), and include a variety of instructional activities such as 
discussion, peer critique/review, group projects, etc. (Leh & Burk, 2016; O’Keefe, Rafferty, Gunder, & 
Vignare, 2020; Roddy, et. al., 2017). In an ERT situation, particular care should be taken to ensure that 
course changes and expectations are clear, that students are able to contact the instructor, and that the 
change of format from either on-campus or blended to fully remote includes communication 
opportunities that are accessible to all students. In both planned online learning experiences and ERT 
environments, the communication focus should remain on creating an open and equitable community, 
and in translating the human elements of communication in teaching and learning to the online 
environment. 
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Technology 


The rapid advancement of educational technology over the past two decades has made it possible to 
provide highly engaging, accessible, and effective online learning experiences to a wide audience of 
students. However, it has also brought an expansive catalog of possibilities that can be challenging to 
navigate when designing an online course, and even more so when shifting to ERT. In either situation, it 
is key to bear in mind that technology should be selected to fit course needs and learning outcomes, 
and the success of technological tools will depend upon their implementation (Hodges, Moore, Lockee, 
Trust, & Bond, 2020; O’Keefe, Rafferty, Gunder, & Vignare, 2020). 


Typically, the learning management system (LMS) is the primary technological tool for delivering online 
or remote instruction. In addition to the LMS, there are many options, both technology and digital 
courseware, that provide additional choices for content delivery, engagement, and assessment. 
Primary considerations for selecting technology tools and digital courseware in both online learning 
and ERT, beyond ensuring that the tools selected meet the needs of the course, include consideration of 
asynchronous and synchronous capabilities, any associated costs to either the institution or student, 
and how the tool will be integrated into the course (via LMS or independently) and supported (O’Keefe, 
Rafferty, Gunder, & Vignare, 2020; Roddy, et al., 2017). In an emergent situation, it is particularly 
important to consider the flexibility needs of students, as well as the potential costs, including financial, 
time and effort to learn the tool(s), and reliable access to the tool(s) (O’Keefe, Rafferty, Gunder, & 
Vignare, 2020). 


Student Support 


Student support is critical to student success in all learning environments, but requires dedicated 
forethought for online learners, as they may not have access to on-campus resources (Roddy et al., 
2017). As Roddy et al. synthesized, there are four key areas for supporting online students: academic 
support, technology assistance, health and well-being resources, and a sense of belonging and/or 
community. 


Academic support includes the aforementioned student-instructor interaction, orientation services, 
appropriate scaffolding for effective online learning, and access to resources such as tutoring and 
library services (O’Keefe, Rafferty, Gunder, & Vignare, 2020; Roddy et al., 2017). Additionally, any 
resources specific to the course should be provided in online-accessible options. Technology assistance 
includes proactive support in selecting user-friendly and flexible options and providing training 
opportunities up front as well as access to ongoing technical support, as needed. Health and well-being 
services for online students can be challenging; there are some barriers with regard to geographical 
location and even funding models for student fees that potentially limit access to such services. 
However, there are options for providing care and referral services, and technology has enhanced the 
availability of such services - through both institutions and third-party providers. It is important for 
faculty to be open, available, and observant of potential needs in order to optimally support student 
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needs for health and well-being services. Finally, establishing belonging and community is considered 
an essential support for online students. Communication among students and peers as well as 
instructors is critical, and it is most effective to both ensure that these interactions are high-quality and 
infused with human connection. 


In an ERT situation, it is not always possible to quickly provide all of these areas of student support. In 
such environments, reviewing critical needs for course success (such as library services/access, for 
example) can guide prioritization, and faculty communication and awareness of indicators of student 
needs are the most important considerations (O’Keefe, Rafferty, Gunder, & Vignare, 2020). 


Faculty Support 


In addition to student support, faculty support is essential for successful experiences in both online 
teaching and learning as well as ERT (Serianni, 2020). The faculty transition to successful online 
teaching and learning includes the acquisition of both pedagogical and technical skills suited to the 
online environment along with an understanding of the needs of online learners (Roddy et al., 2017; 
Serianni, 2020). Though many skills do translate from other modalities to online teaching, some specific 
competencies identified in the literature for online environments include effective communication 
skills, adequate technological/technical skills, and proficiency with monitoring student progress, 
providing student feedback, and appropriate follow-up are essential to reduce student disengagement. 
In addition to these individual needs, it is important for faculty to have access to instructional design 
support and technology support and/or training (O’Keefe, Rafferty, Gunder, & Vignare, 2020). 


In ERT situations, not only do faculty not have the time to engage in support services prior to teaching in 
the online environment, but they may have limited or no access to services that are normally provided 
(O’Keefe, Rafferty, Gunder, & Vignare, 2020). Many organizations and institutions may gather self-serve 
resources in these situations, and in addition to these, faculty should attend to recommendations noted 
previously regarding course design, content, assessment, technology, and support. 
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Higher Education Survey 


Introduction 

Thank you for your participation in this study. The purpose is to conduct a developmental study to 
understand the national position to provide quality online learning across institutions during the 
COVID-19 pandemic in the Kingdom of Saudi Arabia (KSA) and identify areas for future investment 
and improvement. The resulting recommendations will provide information to support capacity and 
quality in online education throughout the kingdom to enhance the potential for Vision 2030 


excellence in a diversified and knowledge-based economy. Please note that participants must be 18 
years of age or older. 


* What best describes your role at your institution? 


© Administrator 


Staff 
Faculty 


Student 


Sm Ce 


Higher Education Survey 


Part | - Leadership - Administrators/Staff 

In this section, questions are designed to understand the perceptions and experiences of 
administrators and/or staff with regard to leadership as related to online education. Please respond to 
these items as they relate to your role as an administrator and/or staff member, and select ’N/A’ for 
any items that do not apply to your role or experience. 


The institution has a governance structure to enable clear, effective, and comprehensive decision making 
related to online education. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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The institution has a governance structure to enable systematic and continuous improvement related to the 
administration of online education. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


, 
f 


The institution has a governance structure in which decisions related to online learning can be reviewed by 
faculty. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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The institution has formed a council that includes a diverse group of stakeholders including faculty, program 
administrators, and support service administrators to support online programs. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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The institution has policies that encourage the development of online courses and programs (e.g., a policy for 
intellectual property of course materials). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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The institution has developed and articulated an institutional strategy for online learning while aligning with 
existing university- and program-level plans (e.g., university strategic plans, university and program 
technology plans, and academic program plans). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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The institution provides leadership and vision for the institutional direction regarding the role of online learning. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
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The institution has a senior level position to lead online learning in academic affairs (e.g., Associate Provost). 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


-. 


The institutional structure provides centralized support for online program development, faculty within online 
programs, and students enrolled in online programs. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


C ‘eo Cc * C & 


The institution has established training and development or faculty development programming to provide 
faculty with the knowledge, skills, and abilities to design and teach online courses. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Higher Education Survey 


Part | - Leadership, Continued - Administrators/Staff 


The institution provides support for faculty, including pedagogical support and instructional training as well as 
technical support, training, and/or production. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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The institution provides centralized and dedicated instructional design support organized under an academic 
reporting structure. 
Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
C) Co C) @ * C) 


The institution has organized the instructional designers with positional parity to faculty through faculty status 
or degrees required. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


e C G & G G 
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The institution has a process for planning and allocating resources for online learning, including financial 
resources, in accordance with strategic planning. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


e G e * e @ 


The institution demonstrates sufficient resource allocation, including financial resources, in order to effectively 
support the mission of online education. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
‘@ ¢ & * ( ( 
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The institution has planned for projected resource needs to scale online learning. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) ¢ C) ® e C) 


Incentives and/or financial compensation are provided to faculty for professional development and course 
design activities. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


e C @ ® ® e 


The institution provides financial incentives and support resources to advance pedagogical and technological 
innovation. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


() i Ci _ eS ( 


Institutions include the aspect of innovation in teaching and learning as part of an award system and/or as part 
of the evaluation and promotion policies (e.g., award for innovation in teaching and learning; promotion review 
includes measure of innovation in teaching). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Curriculum Design and Planning - Administrators/Staff 

In this section, questions are designed to understand the perceptions and experiences of 
administrators and/or staff with regard to curriculum design and planning as related to online 
education. Please respond to these items as they relate to your role as an administrator and/or staff 
member, and select ’N/A’ for any items that do not apply to your role or experience 


Curriculum development is a core responsibility for faculty and program administration (i.e., faculty should be 
involved in either the development or the decision-making for the online curriculum choices). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
a * * rs C1 * 


Guidelines regarding standards for course development, and instructional design and delivery of online 
courses (such as course syllabus elements, course materials, assessment strategies, faculty feedback) are 
used. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


The instructional design of courses is supported by faculty, instructional designers, technical experts, and peer 


evaluators. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


y 


Acourse development process is followed that ensures courses are designed so that students develop the 
necessary knowledge and skills to meet measurable learning outcomes at the course and program level. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


The instructional design focuses on active learning and is student-centered, creating opportunities for 
students to interact with each other and the instructor. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


a ec / 


The instructional design includes a consistent course structure, manageable units, and organization across 
courses for coherency of student experience. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


- — r o~ 
f f 


Instructional materials are accessible to all students, easy to use, and accessible on any Internet-ready 
device. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


f 


Online courses are designed based on accessibility guidelines with consideration for usability, including ease 
of navigation. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Online courses include a syllabus outlining course objectives, learning outcomes, assessment methods, and 
descriptions of learning activities, including course content. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
@ e e e C e 


The institution supports and seeks use of pedagogical and technological innovation methods to improve 
student outcomes, ensure student success, and enhance access and equity through curricular redesign 
programming. 

Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Online Teaching and Learning - Administrators/Staff 

In this section, questions are designed to understand the perceptions and experiences of 
administrators and/or staff with regard to online teaching and learning as related to online education. 
Please respond to these items as they relate to your role as an administrator and/or staff member, and 
select ’N/A’ for any items that do not apply to your role or experience. 


Faculty are expected to hold virtual office hours and frequently communicate with their students via email, 
course news and announcements, online discussions, and assignment feedback. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


O - e Se e e 


Course design promotes student engagement, including academic challenge and social involvement with 
each other and the instructor. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


Online courses have comparable outcomes to onsite courses. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Student-to-student and faculty-to-student interaction are essential characteristics and are encouraged and 
facilitated. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


C * @ e e ° 


Faculty and instructors provide timely and detailed feedback on assessments and student inquiries. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e C O & e O 


The courses resources provide richness in learning materials and activities, support and instruction, instructor 
interactions, and tools and media. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
oO r 4 r a a O) 
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Faculty and instructors are encouraged to innovate in their online teaching and in approaches to student 
learning. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Assessment - Administrators/Staff 

In this section, questions are designed to understand the perceptions and experiences of 
administrators and/or staff with regard to assessment as related to online education. Please respond 
to these items as they relate to your role as an administrator and/or staff member, and select ‘N/A’ for 
any items that do not apply to your role or experience. 


Online courses have multiple timely and appropriate activities to assess student readiness for course content 
and mode of delivery. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) © C) @ s C) 
Frequent and substantial feedback from the instructor is provided to students through assessments. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
© ® CQ) e C e 
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Students are not assessed solely on tests/quizzes but are provided ample opportunity to demonstrate 
proficiency in different ways. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) S 3 & é C) 


Faculty and instructors pilot assessment strategies that incorporate digital literacies and other pertinent 
literacies (e.g.,blogging, visual communication). 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Technology - Administrators/Staff 

In this section, questions are designed to understand the perceptions and experiences of 
administrators and/or staff with regard to technology as related to online education. Please respond to 
these items as they relate to your role as an administrator and/or staff member, and select ‘N/A’ for 
any items that do not apply to your role or experience.n... 


A documented technology strategic plan is in place to ensure quality standards in online learning. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
O) 6 @ ® O C) 
A documented technology operational plan is in place to ensure quality standards in online learning. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


= is O ‘# ‘ CO ; a 


The institution provides adequate resources and support to ensure core learning technology systems are in 
reliably functioning condition and meet operational requirements. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) ¢ (\ r * ( \ (\ 


Support for building and maintaining the online learning technological infrastructure is addressed by a 
centralized system (e.g., Learning Management System) with a single sign-on (SSO) or single identification. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
B r ( \ @ i € 
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The Learning Management System has an interoperability protocol established for applications to be 
embedded within the system for seamless experience. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


Course embedded technology actively supports the achievement of learning outcomes and delivery of course 


content, and superfluous use of technology is minimized. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


The learning management system enables online courses to be delivered asynchronously (over time) and 
synchronously (real-time). dependent on the learning objectives. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


Blended courses are delivered integrating a combination of face-to-face, online, synchronous, and 
asynchronous components. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


Digital security measures are in place to ensure the integrity and validity of information, including identity 
access. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


fo s dg 


IT service management (ITSM) to design, plan, deliver, operate and control information technology support 
the development and delivery of online courses and programs. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


F r f f \ f 
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The institution provides reliable access to wireless internet (WiFi) that is available across the campus, 
including outdoor spaces. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


New and innovative technologies are piloted, supported, and encouraged in online courses on a timely basis 
at the institution. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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A process is followed for evaluating the effectiveness of current and emerging technologies to support the 
achievement of learning outcomes and delivery of course content. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) e O) * é @ 
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Part | - Student Support - Administrators/Staff 

In this section, questions are designed to understand the perceptions and experiences of 
administrators and/or staff with regard to student support as related to online education. Please 
respond to these items as they relate to your role as an administrator and/or staff member, and select 
*’NIA’ for any items that do not apply to your role or experience. 


Students are provided orientations to their online courses, the course design, and the technologies used in the 
courses. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e e e e e O 
Students are provided information on university technology and academic support resources. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


©) C C) * s CO) 


Students have access to appropriate support personnel to address student technical and academic questions, 
problem reporting, and complaints. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
* » O e O O 


Resources are in place to ensure all students, including underrepresented students, have an equitable 
experience (e.g., technology, course materials, support help). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
( \ C" () @ C tC) 
Policy, processes, and resources are in place to support students with disabilities. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e ® e * e e 


Online courses demonstrate compliance with accessibility standards. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
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Students are being supported in innovative ways, such as relevant communication modes or through Al 
communication tools. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Training and Support - Administrators/Staff 

In this section, questions are designed to understand the perceptions and experiences of 
administrators and/or staff with regard to training and support as related to online education. Please 
respond to these items as they relate to your role as an administrator and/or staff member, and select 
‘N/A’ for any items that do not apply to your role or experience. 


Technical assistance is provided for faculty during online course development and online teaching. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


* C e e C C) 


The institution ensures faculty receive training, assistance, and support to prepare for course development 
and teaching online. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


A e 0 co e e 


Faculty are provided ongoing professional development related to online teaching and learning. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
& , C) ry ‘eo CO) 
The institution provides adequate time for faculty to engage in professional development. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e * e e e e 
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Clear standards are established for faculty engagement and expectations concerning online teaching (e.g. 
response time, contact information, etc.). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) e O e OO @ 


Faculty are provided an orientation to technology and instructional design used in the development of online 
courses. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


e « e e e 


There are peer mentoring resources available to faculty members teaching online courses. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


() cr C7 rN eS tj 


Faculty are informed about emerging technologies and the selection and use of new tools. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
O * CQ ¥ C O 
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Higher Education Survey 


Part 1 - Evaluation of Course Outcomes and Program Quality - Administrators and Staff 

In this section, questions are designed to understand the perceptions and experiences of 
administrators and/or staff with regard to evaluation of course outcomes and program quality as 
related to online education. Please respond to these items as they relate to your role as an 
administrator and/or staff member, and select ’N/A’ for any items that do not apply to your role or 
experience. 


The institution has developed a procedure and/or policy for evaluating and improving courses and programs 
on a timely basis. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) & C) * C C) 
Online programs are assessed through an evaluation process that applies specific established standards. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e ® e . e e 


Intended learning outcomes at the course and program level are reviewed regularly to ensure alignment, 
clarity, utility, appropriateness, and effectiveness. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


O * e ® e 6 


A process is in place and followed for the assessment of student retention in online courses and programs. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


e * e e e e 


A variety of data (academic and administrative information) are used to regularly and frequently evaluate 
program effectiveness and to guide changes toward continual improvement. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e C @ e e © 
A process is in place and followed for the assessment of support services for faculty and students. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
© e e * e © 
A process is in place and followed for the institutional assessment of faculty online teaching performance. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


CO) C e © e ©) 


A process is in place and followed for the assessment of stakeholder (e.g., learners, faculty, staff) satisfaction 
with the online courses and programs. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) » C e e @ 


The institution uses innovative developments in analytics to guide course and programmatic decisions about 
online teaching and learning. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
O + e e CG e 
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Part | - Leadership - Faculty 

In this section, questions are designed to understand the perceptions and experiences of faculty with 
regard to leadership as related to online education. Please respond to these items as they relate to 
your role as a faculty member, and select ’N/A’ for any items that do not apply to your role or 
experience. 


The institution has a governance structure to enable clear, effective, and comprehensive decision making 
related to online education. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


The institution has a governance structure to enable systematic and continuous improvement related to the 
administration of online education. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


The institution has a governance structure in which decisions related to online learning can be reviewed by 
faculty. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


o— a. 


The institution has formed a council that includes a diverse group of stakeholders including faculty, program 
administrators, and support service administrators to support online programs. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 


- ro — ro cr 


The institution has policies that encourage the development of online courses and programs (e.g., a policy for 
intellectual property of course materials). 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 


i on ye cn 
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The institution has developed and articulated an institutional strategy for online learning while aligning with 
existing university- and program-level plans (e.g., university strategic plans, university and program 
technology plans, and academic program plans). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


r~ — a _— ~~ 
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The institution provides leadership and vision for the institutional direction regarding the role of online learning. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


pa — <= — — 
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The institution has a senior level position to lead online learning in academic affairs (e.g., Associate Provost). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


C) c oe ; ( () 


The institutional structure provides centralized support for online program development, faculty within online 
programs, and students enrolled in online programs. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


e s e e © e 


The institution has established training and development or faculty development programming to provide 
faculty with the knowledge, skills, and abilities to design and teach online courses. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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\ 


yiball jAjoll OL Ras LEARNING 


Higher Education Survey 


Part | - Leadership, Continued - Faculty 


The institution provides support for faculty, including pedagogical support and instructional training as well as 
technical support, training, and/or production. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e C e * e * 
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The institution provides centralized and dedicated instructional design support organized under an academic 
reporting structure. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
e CS @ e e e 


The institution has organized the instructional designers with positional parity to faculty through faculty status 
or degrees required. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) * Cy @ @ Cy 
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The institution has a process for planning and allocating resources for online learning, including financial 
resources, in accordance with strategic planning. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) * C) e ( @ 


The institution demonstrates sufficient resource allocation, including financial resources, in order to effectively 
support the mission of online education. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) C C) * C) C) 
The institution has planned for projected resource needs to scale online learning. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Incentives and/or financial compensation are provided to faculty for professional development and course 
design activities. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) € ( eS ( ( 1) 


The institution has a governance structure to enable systematic and continuous improvement related to the 
administration of online education. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


e C e & C e 


The institution provides financial incentives and support resources to advance pedagogical and technological 
innovation. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e C O * C OQ 


Institutions include the aspect of innovation in teaching and learning as part of an award system and/or as part 
of the evaluation and promotion policies (e.g., award for innovation in teaching and learning; promotion review 
includes measure of innovation in teaching). 

Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Curriculum Design and Planning - Faculty 

In this section, questions are designed to understand the perceptions and experiences of faculty with 
regard to curriculum design and planning as related to online education. Please respond to these 
items as they relate to your role as a faculty member, and select ’N/A’ for any items that do not apply to 
your role or experience. 


Curriculum development is a core responsibility for faculty and program administration (i.e., faculty should be 


involved in either the development or the decision-making for the online curriculum choices). 
Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
C7 ‘a @ * eS (>) 
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Guidelines regarding standards for course development, and instructional design and delivery of online 
courses (such as course syllabus elements, course materials, assessment strategies, faculty feedback) are 
used. 
Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
() rc {> -s Zz () 
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The instructional design of courses is supported by faculty, instructional designers, technical experts, and peer 
evaluators. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e «* O @ a ~ 


Acourse development process is followed that ensures courses are designed so that students develop the 
necessary knowledge and skills to meet measurable learning outcomes at the course and program level. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


O « * C C O 


Online courses are designed with specific measurable learning objectives and aligned with assessments and 
learning activities. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e C C ® C e 


Instructional materials are accessible to all students, easy to use, and accessible on any Internet-ready 
device. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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The instructional design focuses on active learning and is student-centered, creating opportunities for 
students to interact with each other and the instructor. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
Cy * C) e O C) 


The instructional design includes a consistent course structure, manageable units, and organization across 
courses for coherency of student experience. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) C oe e C) C) 


Online courses are designed based on accessibility guidelines with consideration for usability, including ease 
of navigation. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 


O ® O e e @ 
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Online courses contain orientation materials, including course policies, directions and expectations, and 
instructions for technical and academic support. 

Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Curriculum Design and Planning, Continued - Faculty 
Online courses include a syllabus outlining course objectives, learning outcomes, assessment methods, and 
descriptions of learning activities, including course content. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


O C e # CG e 


Online courses contain clear, concise, and useful assignment instructions. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
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Content, including reading assignments and lectures, aligns with course assessment and learning outcomes. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
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Course content used includes materials that are accessible and interactive, such as Open Education 
Resource (OER). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e C © * # e 


Courses include content that is current, rich, and sufficient in breadth and depth. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) es @ ry C () 
Courses include content that supports dialogue and critical reflection and analysis. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
O C & CO C) 


Content is related to real-word applications. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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There is consistency in course development for student retention and quality. 
Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
O C © * e « 


Faculty provides substantial context to content provided to students. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e * CO ¢ O @ 


The institution supports and seeks use of pedagogical and technological innovation methods to improve 
student outcomes, ensure student success, and enhance access and equity through curricular redesign 
programming. 

Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Online Teaching and Learning - Faculty 

In this section, questions are designed to understand the perceptions and experiences of faculty with 
regard to teaching and learning as related to online education. Please respond to these items as they 
relate to your role as a faculty member, and select ’N/A’ for any items that do not apply to your role or 
experience. 


Faculty are expected to hold virtual office hours and frequently communicate with their students via email, 
course news and announcements, online discussions, and assignment feedback. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
\ = in 
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Course design promotes student engagement, including academic challenge and social involvement with 
each other and the instructor. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
QO * © e O e 


Expectations for student assignment completion, including course schedule with deadlines, grading scheme 
and associated policies, and faculty feedback are provided in the course syllabus. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
e e 0 e C CO 
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Barriers to learning are reduced in the courses by including clear expectations of student activity (participation 
and performance). 
Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
O e O * e 
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Online courses have comparable outcomes to onsite courses. 
Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
O * C e CO) C) 
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Student-to-student and faculty-to-student interaction are essential characteristics and are encouraged and 
facilitated. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
O e e e & O 


Student interactions with other students and with the instructor are facilitated in a variety of ways. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
C) ¢ ( r (% ( ) 


Students participate in meaningful work groups promoting collaboration among students. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e e e é e e 
Peer interaction facilitates active learning through frequent and ongoing peer-involvement. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Feedback on student assignments and questions is constructive and provided in a timely manner. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
© C e ® e e 


Faculty provides timely and detailed feedback on assessments and student inquiries. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
C) 6 C) @ C | C) 


The course resources provide richness in learning materials and activities, support and instruction, instructor 
interactions, and tools and media. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e C O « e e 
Faculty are encouraged to innovate in their online teaching and in approaches to student learning. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Assessment - Faculty 
In this section, questions are designed to understand the perceptions and experiences of faculty with 
regard to assessment as related to online education. Please respond to these items as they relate to 
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your role as a faculty member, and select ’N/A’ for any items that do not apply to your role or 
experience. 


Online courses have multiple timely and appropriate activities to assess student readiness for course content 
and mode of delivery. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


rc rt o> os f ae 
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Ongoing multiple assessment strategies (formative assessment) are used to measure content knowledge, 
skills, and abilities while providing students feedback on what they have learned. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


wo , oi “— ssi a 
r } { 


Students’ self-assessments and peer review and feedback opportunities exist throughout the course. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Learning objectives and assessment activities are closely aligned. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


-_ o™~ eZ r = os 
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Regular feedback about student performance on assessments is provided in a timely manner throughout the 
course. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
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Frequent and substantial feedback from the instructor is provided to students through assessments. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Assessment criteria are clearly described when the assignment is issued with grading rubrics and samples of 
assignments illustrating instructor’s expectations are provided. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Students are not assessed solely on tests/quizzes but are provided ample opportunity to demonstrate 
proficiency in different ways. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


a f- > on 


Faculty pilot assessment strategies that incorporate digital literacies and other pertinent literacies 
(e.g., blogging, visual communication). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


O * O e @ @ 


iboll jSpoll 
1 ONLINE LEARNING 


Higher Education Survey 


Part | - Technology - Faculty 

In this section, questions are designed to understand the perceptions and experiences of faculty with 
regard to technology as related to online education. Please respond to these items as they relate to 
your role as a faculty member, and select ’N/A’ for any items that do not apply to your role or 
experience. 

Core learning technology systems reliably functioning. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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There is a centralized system (e.g., Learning Management System) with a single login. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


© * O) e e O) 


Course embedded technology actively supports the achievement of learning outcomes and delivery of course 
content, and superfluous use of technology is minimized. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


& ) cr C) oN eo C) 


Online courses are delivered asynchronously (over time) and synchronously (real-time) dependent on the 
learning objectives. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
( ) t r ¢ ( * () 


Blended courses are delivered integrating a combination of face-to-face, online, synchronous, and 
asynchronous components. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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The institution provides reliable access to wireless internet (WiFi) that is available across the campus, 
including outdoor spaces. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
O) * O * O O) 


New and innovative technologies are piloted, supported, and encouraged in online courses on a timely basis 
at the institution. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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A process is followed for evaluating the effectiveness of current and emerging technologies to support the 
achievement of learning outcomes and delivery of course content. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Student Support - Faculty 

In this section, questions are designed to understand the perceptions and experiences of faculty with 
regard to student support as related to online education. Please respond to these items as they relate 
to your role as a faculty member, and select ’N/A’ for any items that do not apply to your role or 
experience. 


Students are provided orientations to their online courses, the course design, and the technologies used in the 
courses. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


= ‘a C) o~ = () 


\ i] 
Students are provided information on university technology and academic support resources. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


O * CO e e O 


a \ x 


Students have access to appropriate support personnel to address student technical and academic questions, 
problem reporting, and complaints. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


O) @ * * & () 


Resources are in place to ensure all students, including underrepresented students, have an equitable 
experience (e.g., technology, course materials, support help). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) @ ( ) @ C3 C) 


Policy, processes, and resources are in place to support students with disabilities. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


O C e ® e e 


Online courses demonstrate compliance with accessibility standards. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e s @ r e@ ( } 
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Students are being supported in innovative ways, such as relevant communication modes or through Al 
communication tools. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Training and Support - Faculty 

In this section, questions are designed to understand the perceptions and experiences of faculty with 
regard to training and support as related to online education. Please respond to these items as they 
relate to your role as a faculty member, and select ’N/A’ for any items that do not apply to your role or 
experience. 


Technical assistance is provided for faculty during online course development and online teaching. 
Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
O) s 0) a C O 
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The institution ensures faculty receive training, assistance, and support to prepare for course development 
and teaching online. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Faculty are provided ongoing professional development related to online teaching and learning. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) * C) @ e C) 
The institution provides adequate time for faculty to engage in professional development. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


e C e e O O 


Clear standards are established for faculty engagement and expectations concerning online teaching (e.g. 
response time, contact information, etc.). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Faculty are provided an orientation to technology and instructional design used in the development of online 
courses. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
O * e ¥ C O 


There are peer mentoring resources available to faculty members teaching online courses. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
@ @ (\ = C% () 
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Faculty are informed about emerging technologies and the selection and use of new tools. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
e & @ ® C O 


wi 

a Ze, yiball jSsoll fl i ONLINE LEARNING 
Seana) SOO ol OBC loon sortius 
. BT National eLearning Center 


Higher Education Survey 


Part 1 - Evaluation of Course Outcomes and Program Quality - Faculty 

In this section, questions are designed to understand the perceptions and experiences of faculty with 
regard to evaluation of course outcomes and program quality as related to online education. Please 
respond to these items as they relate to your role as a faculty member, and select 'N/A’ for any items 
that do not apply to your role or experience. 


Online programs are assessed through an evaluation process that applies specific established standards. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) ¢ ( rN C (\ 


Intended learning outcomes at the course and program level are reviewed regularly to ensure alignment, 
clarity, utility, appropriateness, and effectiveness. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


e * e e e e 


Instructional materials and course syllabi are reviewed periodically to ensure they meet online course and 
program learning outcomes. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
O e 0 oO e O) 


7% \. \ \ 


Course assignments and activities are reviewed periodically to ensure they meet online course and program 
learning outcomes. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
© C CO) e e O 
Faculty are expected to participate in peer evaluation of their online courses as well as the courses of others. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
©) C Ci) & e C) 
Course evaluations collect feedback on the effectiveness of instruction and quality of online course materials. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) » C e e ©) 


The institution uses innovative developments in analytics to guide course and programmatic decisions about 
online teaching and learning. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
@ C O * e © 
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Part | - Curriculum Design and Planning - Students 

In this section, questions are designed to understand the perceptions and experiences of students 
with regard to curriculum design and planning as related to online education. Please respond to these 
items as they relate to your role as a student, and select ’N/A’ for any items that do not apply to your 
role or experience. Please respond to the questions in this section based on your Spring 2020 term. 


| was able to develop the necessary knowledge and skills to meet learning outcomes. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


f f 


Courses focused on active learning and were student-centered, creating opportunities for students to interact 
with each other and the instructor. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


Courses had consistent course structure, manageable units, and organization across courses for coherency 
of student experience. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


Pa / 4 —~ ~~ 
f \ f f 


Instructional materials were accessible to all students, easy to use, and accessible on any Internet-ready 
device. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


Online courses were designed based on accessibility guidelines with consideration for usability, including ease 
of navigation. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Online courses included a syllabus outlining course objectives, learning outcomes, assessment methods, and 
descriptions of learning activities, including course content. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
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Instructors helped students understand the importance of course topics and how they are related to learning 
outcomes. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Learning activities facilitated and supported learning that was active, encouraging frequent and ongoing 
engagement with other students. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Curriculum Design and Planning, Continued - Students 


Syllabi were easily located and include objectives, expected outcomes, and completion requirements. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
O ® O € OC O 
Expectations of student participation (frequency and quality) were included in the syllabi or online. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
© e C) e e © 
Instructional materials had sufficient breadth, depth, and currency for me to learn the subject. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
O * e s O O 


The materials included current online materials (online articles, webpages, links, and/or videos). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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The class was organized within the learning management system in a way that was easy to navigate. 
Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
O s O ® O OC) 
The class was accessible. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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The design of the course reflected innovative thought about teaching and learning in an online environment. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
@ s e ° « e 


Online courses included innovative teaching methods and technologies to promote student equity, learning, 
and academic achievement. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Online Teaching and Learning - Students 

In this section, questions are designed to understand the perceptions and experiences of students 
with regard to online teaching and learning as related to online education. Please respond to these 
items as they relate to your role as a student, and select ’N/A’ for any items that do not apply to your 
role or experience. Please respond to the questions in this section based on your Spring 2020 term. 


Faculty held virtual office hours and frequently communicated with students via email, course news and 
announcements, online discussions, and assignment feedback. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


e ' 6 e C e 


Students communicated with other classmates through online discussions. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) * C) @ 6 C) 


Faculty provided ongoing and meaningful communication in a timely, effective, and appropriate manner. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
© C O # C CO 
Students were academically challenged and interacted with each other and the faculty. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
O c O * ¢ O} 


Course learning outcomes were clearly defined and linked with activities and assessments. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
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Instructions on how to meet the expected outcomes were adequate and stated clearly. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
(C) Bi C\ € ‘ (\ C \ 

Online courses had comparable outcomes to onsite courses. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
O C O @ e e 


Expectations for student assignment completion, including course schedule with deadlines, grading scheme 
and associated policies, and faculty feedback were provided. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
O * @ & CG ° 


Courses clearly defined student activity expectations (participation and performance) to reduce barriers 
related to online learning. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Online Teaching and Learning, Continued - Students 


Student-to-student and faculty-to-student interactions were encouraged and facilitated in online courses. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) ® * & CC CO) 


Peer interaction facilitated active learning through frequent and ongoing peer-involvement. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Students meaningfully interacted with course content beyond course lectures through groups, online 
discussions, activities, and assessments. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
(| SS ( * é C) 


Faculty and instructors provided ongoing, meaningful, and constructive feedback in a timely, effective, and 
appropriate manner. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


e « O * C e 


Faculty and instructors provided timely and detailed feedback on assessments and student inquiries. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


CO e @ oe * O 


Course resources provided richness in learning materials and activities, support and instruction, instructor 
interactions, and tools and media. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Students had access to resources necessary to succeed in an online course. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Faculty and instructors regularly innovate their online teaching. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Assessment - Students 

In this section, questions are designed to understand the perceptions and experiences of students 
with regard to assessment as related to online education. Please respond to these items as they relate 
to your role as a student, and select ’N/A’ for any items that do not apply to your role or experience. 
Please respond to the questions in this section based on your Spring 2020 term. 


Students are assessed on their readiness for online learning. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


r = re aa r cr 


Assessment criteria were clearly described when the assignment is issued with grading rubrics and samples 
of assignments illustrating instructor’s expectations. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


CY f f 


Online courses had multiple timely and appropriate activities to assess student learning. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
, , (— 2 -& = Zz 


Avariety of assessments were provided to enable different learners opportunities to demonstrate skills. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


f 4 a 


Course assessments were provided in different formats (quizzes, exams, short response papers) 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Frequent and substantial feedback from the instructor was provided to students through assessments. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


, f- va - c~ 
f f 


Students were not assessed solely on tests and quizzes but are provided ample opportunity to demonstrate 
proficiency in different ways. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Students had regular opportunities to self-assess their course progress, identify opportunities for 
improvement, and set goals for personal and professional growth. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Students had opportunities to learn from their peers through peer evaluation of one another’s work and other 


activities. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
ea * C) e Oi C) 


My overall course grade was not based solely on exams and quizzes. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


e C O e e O} 


New assessment methods are regularly introduced to measure content knowledge, skills, and abilities in 
online learning. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Courses include assessment strategies that incorporate digital literacies and other pertinent literacies (e.g., 
blogging, visual communication). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Technology - Students 


In this section, questions are designed to understand the perceptions and experiences of students 
with regard to technology as related to online education. Please respond to these items as they relate 
to your role as a student, and select ’N/A’ for any items that do not apply to your role or experience. 
Please respond to the questions in this section based on your Spring 2020 term. 


| have a computer or laptop. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


e % e E e e 


| have internet in my home or in my dormitory. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
() & ( ) & (3 ( ) 
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The institution provided reliable access to wireless internet (WiFi) that is available across the campus, 
including outdoor spaces. 


Strongly disagree Disagree Neutral Agree Strongly Agree N/A 


| have internet available anytime to work online. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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It was easy for me to log-in to my courses. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


The learning technologies for my courses were reliable and available when | needed to use them. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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| was able to access the Learning Management System (LMS) with a single sign-on. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


ra 


| was able to access all the things | needed for my online learning in one place. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


, ,- va va o~ 
f f 


Online courses were delivered asynchronously (over time) and synchronously (real-time) dependent on the 
learning objectives. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Blended courses are delivered integrating a combination of face-to-face, online, synchronous, and 
asynchronous components. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


The course sites included embedded technologies and functionality that supported learning. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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| feel that my information in my courses was secure. 


Strongly Disagree Disagree Neutral Agree Strongly Agree NIA 
C) * C) @ 6 C) 


My course included new and innovative technologies. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Part | - Student Support - Students 

In this section, questions are designed to understand the perceptions and experiences of students 
with regard to student support as related to online education. Please respond to these items as they 
relate to your role as a student, and select ’N/A’ for any items that do not apply to your role or 
experience. Please respond to the questions in this section based on your Spring 2020 term. 


Students were provided orientations to their online courses, the course design, and the technologies used in 
the courses. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
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Students were provided information on university technology and academic support resources. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 


O C e e O 0) 


Students have access to appropriate support personnel to address student technical and academic questions, 


problem reporting, and complaints. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) ® eo @ @ C) 


\. 


Resources were in place to ensure all students, including underrepresented students, have an equitable 
experience (e.g., technology, course materials, support help). 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) cr C) rN '@ () 
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Resources were in place to support students with disabilities. 


Strongly Disagree Disagree Neutral Agree Strongly Agree 
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Online courses demonstrate compliance with accessibility standards. 


Strongly Disagree Disagree Neutral Agree Strongly Agree 
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Students are being supported in innovative ways, such as relevant communication modes or through Al 


communication tools. 
Strongly Disagree Disagree Neutral Agree Strongly Agree 


( ) ) 0 


How frequently did you receive support from the following and how helpful was that support you received? 


N/A 


N/A 


N/A 


How frequently did you receive support? How helpful was the support? 
Campus Tutoring | $ + | 
Program Tutoring + $ 
Campus Library f ny | ¢| 
Campus Tech Help Desk : ry [ j 


Program Tech Help = 


Third-Party Technology 
Companies 


Advisor 


Classmates 


Friends 


Family 


Lae 


Course Faculty 


Other Faculty 


a> 


Other (please specify) 
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Part 1 - Evaluation of Course Outcomes and Program Quality - Students 

In this section, questions are designed to understand the perceptions and experiences of students 
with regard to evaluation of course outcomes and program quality as related to online education. 
Please respond to these items as they relate to your role as a student, and select ’N/A’ for any items 
that do not apply to your role or experience. Please respond to the questions in this section based on 
your Spring 2020 term. 


Students were provided an opportunity to give feedback on opportunities for course improvement(s) during or 
after the class was completed. 


Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
O e e * C e 


Course evaluations were conducted to collect feedback on the effectiveness of instruction and quality of online 
course materials. 
Strongly Disagree Disagree Neutral Agree Strongly Agree N/A 
C) 6 C\ e C\ C) 
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Part Il - Leadership 

Institutions have gone through tremendous amounts of change. In order to better understand the 
impact of change on different functions of the instruction to support online learning, please rate 1.) 
your satisfaction with each of the institutional functions (e.g., leadership) prior to COVID-19 and 2.) 
your satisfaction with each of the institutional functions *after* COVID-19 influenced the state of online 
learning. 


The leadership is accountable to a governance body and is responsible for setting and meeting the 
operational and strategic goals in support of the online education mission and vision statements. 


Please rate your satisfaction with each of the institutional functions prior to COVID-19. 


Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied NIA 
Governance @ ‘ C * é i. O e e 
Strategies C ‘ € ~ * Cc) e ry 
Policies 2 ( \ rs CO) e 
Processes C CO i. 0 oO O 
Resource Allocation @ (~ rN () 2 
Period Review and 
Updating a © bs O Lo 
Innovation ¢ ® oO O e e 


Please rate your satisfaction with each of the institutional functions after COVID-19 influenced the state of 
learning. 


Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied NIA 
Governance = C- ® C1 € Ci 
Strategies & ae ‘’ ©) ] a: ( | 
Policies e ( ' @ () & ' 
Processes C3 C ‘e ; C C C \ 
Resource Allocation g & et C) C C\ 
aaa ias and ‘eo oe las ( > eo C i 
Innovation = C~ e C) c * Ci 
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Part Il - Curriculum Design & Planning 

A quality online course and/or program adopts and implements instructional design methods that 
enable effective online instruction for both institutionally developed courses as well as licensed 
content from other sources. 
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Please rate your satisfaction with each of the institutional functions prior to COVID-19. 


Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied NIA 
Instructional Design 
Wetdiedte & Uitwienect C 4 oe eo e e a 
Design for Learning . > - . tS 
(UDL) 
Alignment with co ‘ co \ oo \ 
Standards ‘ C \ “S O 
Course Syllabi = (| r’ ( } & ( 
Course Materials & ra . co la oO 

) { 1 

Content b a ? : c 
Innovation # t & C) ‘e ‘@ 


Please rate your satisfaction with each of the institutional functions after COVID-19 influenced the state of 


learning. 

Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied NIA 
Instructional Design 
Methods & Universal Co [™o ao 
Design for Learning Y * = Ci \ @ 
(UDL) 
Alignment with ro ~\ rx = rn _ 
Standards s C ’ O \ C 
Course Syllabi = (— r C) s e 
Course Materials & [7 > o- an co a 
Content - Me ms 2 LL 
Innovation @ a * a e ‘ea 
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Part Il - Online Teaching and Learning 
Online instructors support learning and facilitates presence (teacher, social, and learner) with digital 
pedagogy, communication, and engagement. 
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Please rate your satisfaction with each of the institutional functions prior to COVID-19. 


Very Unsatisfied 

Communication & 
Engagement C3 
Expectation Setting e 
Outcomes C 

Course Interaction & 
Feedback GC 
Resources ® 
Innovation ¢ 4 


Please rate your satisfaction with each of the institutional functions after COVID-19 influenced the state of 


Very Unsatisfied 


learning. 
Communication & 
Engagement ‘+ 
Expectation Setting & 
Outcomes C3 
Course Interaction & 
Feedback CS 
Resources cr 
Innovation ‘<é 


Part Il - Assessment 


Online instructors create and/or implement assessments in online learning environments in ways that 
ensure the validity and reliability of the instruments and procedures. Instructors measure learner 
progress through assessments, projects, and assignments that meet standards-based learning goals, 
and evaluate learner understanding of how these assessments measure achievement of the learning 


objectives. 


Unsatisfied Neutral Satisfied 
C e e 
= ‘+ C) 
C a C) 
C ‘< O 
C° € C ) 
C = O 
* e * 
i C3 eo 


Unsatisfied Neutral Satisfied 
® é 6 
* Ci a 
e os O 
O C O 
C & e 
C C Cy 
e # e 
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Please rate your satisfaction with each of the institutional functions prior to COVID-19. 


Assessment Strategies 
Assessment Processes 


Assessment 
Methodology 


Assessment Types 


Innovation 


Please rate your satisfaction with each of the institutional functions after COVID-19 influenced the state of 


learning. 


Assessment Strategies 
Assessment Processes 


Assessment 
Methodology 


Assessment Types 


Innovation 


Very Unsatisfied 


Very Unsatisfied 


Unsatisfied 


Unsatisfied 


e 


C 

C 
a 
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Neutral 
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Satisfied Very Satisfied 
C) CC 
O o 
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Satisfied Very Satisfied 
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Part Il - Technology 


Educational technology is supported, reliable, accessible, and sufficient to meet the online education 


needs of faculty members and students. 


Please rate your satisfaction with each of the institutional functions prior to COVID-19. 


Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied NIA 
Operability @ ® & C\ € Cy 
Centralized Online \ \ \ 
Education Infrastructure a C © © C © 
Modality s ( = O e @ 
Security € : ( ‘a * (> f (% 
IT Service Management . ; 
Compliance y * * @ 6 e ® 
Internet Access ~ ee © ‘@ ‘* ‘eo 
Reliability & & cr” (\ * é 
Coverage C) C C C1 C3 CO 
Innovation é C 2 OO 2 @ 


Please rate your satisfaction with each of the institutional functions after COVID-19 influenced the state of 
learning. 


Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied NIA 
Operability & (7 * a e e 
beh dit nm ‘@ C ie O * CO 
Modality = C € ( 5 C . C 
Security (> C CG 0 Ee r 
IT Service Management = = u 
Compliance : 3 @ 4 C } a C 
Internet Access c™ 4 a ‘oe C) 'e oO 
Reliability ® C 4 O * e 
Coverage C : C ‘eo @ « ‘e 
Innovation q ‘ 4 e O ~ a 
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Part Il - Student Support 
Student support services are available to address the various needs of learners at different levels 
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within the organization. The levels of support are appropriate and adequate for learner success. 


Please rate your satisfaction with each of the institutional functions prior to COVID-19. 


Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied NIA 
Student Orientation and rr e ry e r '~ 
Support \ N . \ . 
Equity © CO e OQ s O 
Accessibility fe (~ * e e e 
Compliance Standards ‘ O Oo O O O 
Innovation & & — Cy a * 


Please rate your satisfaction with each of the institutional functions after COVID-19 influenced the state of 


learning. 

Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied NIA 
Student Orientation and rx e e e e e 
Support : : : 
Equity C C ‘- O Cc CO 
Accessibility @ & - C) e C 
Compliance Standards Cc CO ‘. C) Cs CO 
Innovation & Ss r O & O 
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Part Il - Training and Support 
Training and support is available to faculty and staff though mentoring, technical assistance, and 
timely professional development. 
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Please rate your satisfaction with each of the institutional functions prior to COVID-19. 


Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied NIA 
Technical Assistance @ & * O € e@ 
Professional oO = oO oO oO O 
Development - : X \ 
Orientation 2 ( f~ C) e eo 
Mentoring i O C * ( \ C . Oy 
Innovation & C & C) a O 


Please rate your satisfaction with each of the institutional functions after COVID-19 influenced the state of 
learning. 


Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied N/A 
Technical Assistance @ C : ® s 5 C “ g \ 
Professional cr am co a cy 7 
Development s ‘ \ 
Orientation € (* € C) € (\ 
Mentoring Cy ‘e ‘eo C O e 
Innovation & 'e & e @ (> 
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Part Il - Evaluation and Continuous Improvement 

Evaluation is both internal and external and informs all processes that affect teaching and learning. 
Internal evaluations often are more informal in nature and may provide immediate feedback on a 
targeted area of inquiry. External program evaluations typically look at the entire program from an 
objective perspective that will bring additional credibility to the results. 
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Please rate your satisfaction with each of the institutional functions prior to COVID-19. 


Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied NIA 
Student Satisfaction @ (~ @ C) £ C 
Faculty Satisfaction C (~ ie CO) i. - 
Staff Satisfaction € (> r () = ( 
Evaluation of Course _ 
Outcomes and Program <= C eS CO) C (\ 
Quality 
Innovation € ( s C c* ( 
Please rate your satisfaction with each of the institutional functions r COVID-19 influen h f 
learning. 
Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied N/A 
Student Satisfaction & (~ & CO t C* 
Faculty Satisfaction C3 ™ ra C) ‘e oy 
Staff Satisfaction t 2 & C) & CC 
Evaluation of Course 
Outcomes and Program ‘oe C) a C C Cy 
Quality 
Innovation r C s O) oO e 
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Part Ill - Challenges 
In this section, please select each potential challenge item as you have perceived or experienced it as 
related to online learning. 
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What are the most pressing challenges associated with online teaching and learning at your institution? 
Check all that apply. 


Quality of Online Courses 


Availability of Professional Development Funding 


Time Management 


Student Online Readiness 


Flexibility of Student Academic Support Services 


Decision-Making Processes 


Area-Specific Leadership 


Familiarity with Online Learning 


Online Work Skills Knowledge (e.g., Microsoft Word) 


Technological Infrastructure 


Course Variety 


Time to Completion 


Instructional Technology Support 


Access to Hardware 


Access to Software 


Access to Internet 


Spaces for Learning 


Innovative Thought Leadership 


Communication with Faculty 


Cost of Course Materials 


Student Engagement with Course 


Faculty-Student Engagement 


Peer-Peer Engagement 


Pedagogical Innovation/Knowledge 


Accessibility/Universal Design Resources 


Shared Governance Policies/Processes 


Opportunities for Peer Learning 


Extra-Curricular Group and Campus Engagement 
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Part Ill - Opportunities for Improvement 
In this section, please rank each potential opportunity for additional resource investment as you have 
perceived or experienced it as related to online learning. 


Where would you recommend an investment of resources related to online teaching and learning? Rank in 
order of importance with the top being the highest. 


‘ol Hl 


+ 
v¥ 


Leadership 


7 
vv 


al Ml 


Institutional Policies 


7 
4 


Faculty/Instructor Training 


+| 
¥ 


Student Academic Support Services 


. 
vv 


Student Online Readiness 


lll 


| 
¥ 


Instructional Design 


a 


Student Technology Resources 
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Faculty Technology Resources 


a 
aA 


Online Work Skills (e.g., Microsoft Word knowledge) 


a 
¥ 


Mental Health Services 


“lll 


a 
v 


Physical Health Services 


a 
vv 


Open Education Resources 


a> 


Innovative Thought Leadership 


a 
vv 


Governance Policies and Processes 


- 
v 


Opportunities for Peer Learning 
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Part IV - Demographics - Administrators/Staff 


What is your gender? 
() Male 


() Female 


What is the name of your institution? 


| 


What branch (if applicable)? 


What type is your institution? 


() Public 


( ') Private 


Position 


O President 

( ) Vice President 
() Dean 

( ) Vice Dean 
(>) Staff 


() Other (please specify) 


Do you hold the position of dean of e-learning in your institution or its equivalent? 
C) Yes 


() No 


93200 eee 


How many years have you been in your current position? (e.g., "5") 
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Part IV - Demographics - Faculty 


What is your gender? 


( >) Male 


() Female 
Nationality 
() Saudi 


() Non-Saudi 


What is the name of your institution? 


What branch (if applicable)? 


Po 


What type is your institution? 


() Public 


() Private 
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Position 


() Professor 
C >) Associate Professor 
( ) Assistant Professor 
() Lecturer 
C ») Teaching Assistant 


() Other (please specify) 


Specialization 


() Human Sciences 

(_.) Natural Sciences and Engineering 
( *) Information and Computer Sciences 
( ) Medical and Health Science 


() Other (please specity) 


Years of Experience 


() Syears or less 
( )' More than 5 years to 10 years 


() More than 10 years to 15 years 


() More than 15 years 


How many years have you been in your current institution? (e.g., “10") 


SS a Ss 


How many years of experience do you have in e-learning? (e.g., "5") 
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What is your gender? 
O Male 
© Female 


Age 

() 18-22 
©) 23-27 
() 28-32 
() 32-36 
C) 36+ 


Region 


City 


What is the name of your institution? 


Po 


What branch (if applicable)? 


Pe 


What type is your institution? 


O Public 
O Private 


Degree 

O Diploma 

O Bachelors 
O Higher Diploma 
2 Masters 

O Doctoral 
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Specialization 


() Human Sciences 

O Natural Sciences and Engineering 
OC Information and Computer Sciences 
QO Medical and Health Science 


O Other (please specify) 


Level 


O Orientation 


O Level 1 


Level 2 


C) 
WW 


Level 3 


Level 4 


Level 5 


Level 6 


Level 7 


Level 8 


Level 9 


Level 10 


OO0O0O0000 


Have you taken an online course before COVID-19? 
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Appendix VIII: Interview Protocols 
Administrator Interview Protocol 


Interviewer: 


Post Interview Comments: 


INSTRUCTIONS 
We are conducting research for the National eLearning Center (NELC), that is intended to better understand 
the state of online learning within the Kingdom of Saudi Arabia pre-COVID-19 and during COVID-19. A survey 


has also been sent out to higher education administrators, staff, faculty and students to obtain their 
perspectives on several dimensions of online learning. These interviews are being used to allow us to seek 
greater depth of information in specific areas. 


Interviewee Background Information 


Interviewee (Title and Name): 
Institution: 
Does your role currently have oversight for any online learning at your institution? Yes / No 
If yes, please describe the extent of that oversight (e.g., institution-wide, department level, 


course-level, course design, etc.) 


How long have you been in your present position? 
..at this institution? 


What is your highest degree earned? 
What was your field of study? 


Did your institution offer online courses or full programs online prior to COVID? 
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Dimension |: Leadership 


Is online learning currently included in your institutional strategic plan? 


If yes... 

la. Please describe your strategy for online learning. 

1b. Do you feel that your institution’s response to the pandemic was aligned to your online 
learning strategy? 


If no... 

1c. Do you anticipate adding a strategy for online learning post-COVID? Why or why not? 

1d. What are the challenges in getting the e-learning agenda going pre-COVID-19? During Covid- 
19? What is your anticipation for the future? 


Do you believe that your institution has sufficient resources (financial, technology, personnel, etc.) to 
support online learning? Why or why not? 


2a. How did this change due to COVID? 
2b. What resources do you believe will be needed to support online learning post-COVID? 


Dimension II: Curriculum Design and Planning 


What methods did your institution use pre-COVID to determine course quality? 


3a. If your institution offered online courses, what methods were used pre-COVID to determine 
the quality of your online courses? 
3b. What changes did you make during the transition to remote learning to assess the quality of 


your courses? 
3c. What methods do you think your institution will need to use post-covid to determine the 


quality of online courses? 


Pre-COVID did your institution provide opportunities for pedagogical and technological innovation 
methods to improve student success? 
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If yes... 
4a. Please share examples of how pedagogical or technological innovation methods were used 


to support student success? 


If no... 


4b. What changes do you believe are needed at your institution to encourage the 
implementation of innovative pedagogical and technological methods? 


Dimension V: Technology 
Pre-COVID did your institution have a technology plan? 


If yes... 

8a. Was the technology plan sufficient to support the needs of your institution during the 
transition to remote learning? Why or why not? 

8b. What changes do you believe will need to be made to it to better support your institution 
post-COVID? 


If no... 


8c. Does your institution have plans to develop a technology plan post-COVID? 
8d. What do you think needs to be included in your technology plan? 


Dimension VI: Student Support 


Pre-COVID what options were provided to students to access support services at your institution? 


During the transition to remote learning, what changes were made by your institution to provide 
students with adequate access to support services? 


What changes does your institution anticipate making post-COVID to ensure all students are able to 
access support services regardless of the modality (in-person, online, blended) of their program? 


Dimension VII: Training and Support 


Please describe the professional development options provided to faculty pre-COVID. 


9a. Did your institution provide training on how to develop quality online courses? 
9b. Did your institution provide training on how to teach quality online courses? 


What professional development did your institution provide to faculty during the transition to remote 
learning? 


What changes does your institution anticipate making post-COVID to prepare faculty to teaching quality 
online courses? 
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Dimension VIII: Evaluation of Course Outcomes & Program 
Quality 


Pre-COVID what data did your institution evaluate to determine the overall quality of a program? 
What additional data did you review to determine quality during the transition to remote learning? 


What changes does your institution anticipate making post-COVID in determining the process needed to 
assess program quality? 
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Faculty Interview Protocol 


Interviewer: 


Post Interview Comments: 


INSTRUCTIONS 

We are conducting research for the National eLearning Center (NELC), that is intended to better understand 
the state of online learning within the Kingdom of Saudi Arabia pre-COVID-19 and during COVID-19. A survey 
has also been sent out to higher education administrators, staff, faculty and students to obtain their 
perspectives on several dimensions of online learning. These interviews are being used to allow us to seek 


greater depth of information in specific areas. 


Interviewee Background Information 


Interviewee (Title and Name): 

Institution: 

Does your role currently have oversight for any online learning at your institution? Yes / No 

If yes, please describe the extent of that oversight (e.g., institution-wide, department level, course-level, 


course design, etc.) 


How long have you been in your present position? 
..at this institution? 


What is your highest degree earned? 
What was your field of study? 


Did your institution offer online courses or full programs online prior to COVID? 
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Dimension |: Leadership 


Is online learning currently included in your institutional strategic plan? 


If yes... 

la. Please describe your strategy for online learning. 

1b. Do you feel that having a strategy for online learning improved your institution’s 
responsiveness during the recent pandemic? 


If no... 
1c. Do you anticipate adding a strategy for online learning post-COVID? Why or why not? 


Do you believe that your institution has sufficient resources (financial, technology, personnel, etc.) to 


support online learning? Why or why not? 


2a. How did this change due to COVID? 
2b. What resources do you believe will be needed to support online learning post-COVID? 


Dimension II: Curriculum Design and Planning 


What methods did your institution use pre-COVID to determine course quality? 


3a. If your institution offered online courses, what methods were used pre-COVID to determine 
the quality of your online courses? 

3b. What changes did you make during the transition to remote learning to assess the quality of 
your courses? 

3c. What methods do you think your institution will need to use post-covid to determine the 
quality of online courses? 


What is your perceptions about online learning? 


Probing questions... 
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4a. Do you enjoy teaching online? 

4b. Do you feel that students are able to learn in this environment? 

4c. Do you feel that this requires more or less effort by faculty teaching in this environment? 
Why? 


Pre-COVID did your institution provide opportunities for pedagogical and technological innovation 
methods to improve student success? 


If yes... 

4a. Please share examples of how pedagogical or technological innovation methods were used 
to support student success? 

If no... 


4b. What changes do you believe are needed at your institution to encourage the 
implementation of innovative pedagogical and technological methods? 


Dimension III: Online Teaching and Learning 


If your institution offered online courses pre-COVID, what methods did you use to encourage student 
engagement in the online course? 


What methods did you use during the transition to remote learning to encourage student engagement 
in their courses? 


What methods do you believe need to be implemented post-COVID to encourage student engagement 
in their courses? 


Dimension IV: Assessment 


Pre-COVID what methods for assessing student learning did you use in your classes? 
During the transition to remote learning, how did you assess student learning? 


Post-COVID what changes do you anticipate you will make to the methods you use to assess student 
learning? 


Dimension VII: Training and Support 


Please describe the professional development options available to faculty pre-COVID? 
11a. Did your institution provide training on how to develop quality online courses? 
11b. Did your institution provide training on how to teach quality online courses? 


11c. Did your institution provide adequate support (SMEs, IDs, Developers, etc.) to help faculty 
design a quality online course? 
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What professional development was provided to you during the transition to remote learning? 


What additional training do you believe is needed post-COVID to prepare faculty to teach quality online 
courses? 


ye 


Student Interview Protocol 


Interviewer: 


Post Interview Comments: 


INSTRUCTIONS 

We are conducting research for the National eLearning Center (NELC), that is intended to better understand 
the state of online learning within the Kingdom of Saudi Arabia pre-COVID-19 and during COVID-19. A survey 
has also been sent out to higher education administrators, staff, faculty and students to obtain their 
perspectives on several dimensions of online learning. These interviews are being used to allow us to seek 


greater depth of information in specific areas. 


Interviewee Background Information 


Interviewee (Title and Name): 

Institution: 

Classification: Freshman, Sophomore, Junior, Senior, Graduate Masters / Doctoral 
How long have you been at your institution? 

Did you transfer from another institution? Y / N 

What is your major? 


Did your institution offer online courses or full programs online prior to COVID? 
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Dimension I: Leadership 


Do you think that your institution provides online learning in a planned and/or strategic way? 


If yes... 

1a. Do you feel that your institution’s response to the pandemic was aligned with the online 
learning strategy? 

1b. Do you feel that your institution’s online strategy provided a smooth transition during 
COVID-19? 


If no... 
1c. Do you feel that your institution’s response to COVID-19 will result in an online learning 
strategy? 


All 
1d. What are the challenges in online learning at your institution pre-COVID-19? During Covid- 
19? What is your anticipation for the future? 


Do you believe that your institution has sufficient resources (financial, technology, 
personnel, etc.) to support online learning? Why or why not? 


2a. How did this change due to COVID-19? 

2b. What resources do you believe will be needed to support online learning in Fall 2020? 

2c. What resources do you believe will be needed to support online learning long-term, after Fall 
2020? 


Are the processes for the following effective? 
3a. Finding and enrolling in online courses 
3b. Accessing online courses 
3c. Accessing technical assistance for online courses 
3d. Accessing academic resources for online courses 
3e. Accessing other support resources for online students 
3f. Communication from your institution about online learning related to COVID-19 


What innovations has your institution implemented in online learning? 


What innovations would you like for your institution to implement in the future? 
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Appendix IX: Pressing Challenges 


Challenges Percentage 


Quality of Online Courses 50.20% 
Availability of Professional Development Funding 34.50% 
Time Management 34.60% 
Student Online Readiness 61.90% 
Flexibility of Student Academic Support Services 31.50% 
Decision-Making Processes 25.00% 
Area-Specific Leadership 16.50% 
Familiarity with Online Learning 43.20% 
Online Work Skills Knowledge 27.70% 
Technological Infrastructure 38.10% 
Course Variety 25.70% 
Time to Completion 26.70% 
Instructional Technology Support 27.70% 
Access to Hardware 27.40% 
Access to Software 26.70% 
Access to Internet 44.40% 
Spaces for Learning 18.40% 
Innovative Thought Leadership 23.70% 
Communication with Faculty 25.70% 
Cost of Course Materials 15.20% 
Student Engagement with Course 37.30% 
Faculty-Student Engagement 34.40% 
Peer-Peer Engagement 27.00% 
Pedagogical Innovation/Knowledge 25.20% 
Accessibility/Universal Design Resources 20.50% 
Shared Governance Policies/Processes 19.10% 
Opportunities for Peer Learning 26.50% 
Extra-Curricular Group and Campus Engagement 33.40% 
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Appendix X: Student Support 


Individual & 

Group Support 
Campus 
Tutoring 
Program 
Tutoring 
Campus Library 


Campus 
Technology 
Help Desk 
Program 
Technology 
Help Desk 
Third-Party 
Technology 
Companies 
Advisor 


Classmates 
Friends 

Family 

Course Faculty 
Other Faculty 


36.10% 


36.60% 


39.70% 
33.60% 


32.90% 


57.70% 


51.90% 
8.30% 

11.00% 
16.40% 
11.00% 
31.10% 


14.50% 


14.80% 


15.80% 
17.00% 


15.60% 


10.80% 


12.10% 
9.60% 
9.10% 
11.20% 
13.80% 
18.70% 


Sometimes 


23.50% 


21.30% 


19.30% 
20.90% 


21.80% 


13.50% 


15.80% 
23.90% 
20.20% 
16.50% 
26.80% 
21.30% 


11.80% 


12.20% 


10.10% 
11.80% 


12.50% 


7.30% 


8.20% 

22.20% 
17.30% 
12.20% 
19.80% 
11.90% 


14.10% 


15.10% 


15.10% 
16.70% 


17.20% 


10.70% 


12.00% 
35.90% 
42.40% 
43.70% 
28.60% 
16.90% 


Not at All 


32.60% 


33.40% 


38.00% 
31.80% 


30.20% 


54.50% 


49.10% 
8.20% 

10.60% 
14.30% 
10.60% 
29.40% 


Slightly 


20.10% 


19.20% 


19.20% 
20.20% 


21.20% 


14.00% 


14.80% 
15.40% 
13.90% 
13.90% 
20.20% 
21.00% 


Moderately 


22.60% 


23.10% 


18.90% 
20.20% 


21.70% 


14.60% 


15.30% 
24.90% 
20.50% 
16.80% 
26.20% 
21.80% 


12.80% 


12.40% 


11.40% 
12.80% 


12.10% 


7.20% 


9.20% 

21.90% 
20.00% 
15.10% 
18.50% 
11.80% 


Extremely 


11.80% 


11.90% 


12.50% 
15.00% 


14.80% 


9.70% 


11.50% 
29.50% 
35.00% 
39.90% 
24.40% 
16.10% 
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Appendix XI: List of Figures 


Dimension |: Leadership 


Governance: Figures 1-4 

Strategies: Figures 5, 6 

Policies: Figures 7, 8 

Process: Figures 9, 10 

Resource Allocation: Figures 11-15 
Periodic Review & Updating: Figure 16 
Innovation: Figures 17-19 


Dimension II: Curriculum Design & Planning 


Instructional Design Methods & UDL: Figures 20-21 
Alignment with Standards: Figures 22, 23 

Course Syllabi: Figures 24, 25 

Course Materials & Content: Figure 26 

Innovation: Figure 27 


Dimension III: Online Teaching & Learning 


Communication: Figure 28 

Engagement: Figure 29 

Expectation Setting: Figure 30 

Outcomes: Figures 31, 32 

Course Interaction: Figure 33 

Feedback: Figure 34 

Teaching & Learning Resources: Figures 35, 36 
Innovation: Figures 37, 38 


Dimension IV: Assessment 


Assessment Strategies: Figure 39 
Assessment Processes: Figures 40-42 
Assessment Methods: Figures 43-46 
Assessment Types: Figure 47 
Innovation: Figure 48 
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Dimension V: Technology 


Centralized Infrastructure: Figure 49 
Internet Access/Reliability: Figures 50-52 
ITSM Compliance: Figure 53 

Modality: Figure 54, 55 
Operability/Security: Figures 56, 57 
Coverage: Figures 58-60 

Innovation: Figures 61, 62 


Dimension VI: Student Support 


Student Orientation & Support: Figures 63-66 
Equity: Figure 67 

Accessibility: Figure 68 

Compliance Standards: Figure 69 

Innovation: Figure 70 


Dimension VII: Training & Support 


Technical Assistance: Figures 71, 72 
Professional Development: Figures 73-77 
Orientation: Figures 78, 79 

Mentoring: Figures 80, 81 

Innovation: Figures 82, 83 


Dimension VIII: Evaluation & Continuous Improvement 


Evaluation of Course Outcomes & Program Quality: Figures 84-86 
Student, Faculty, & Staff Satisfaction: Figures 87-90 
Innovation: Figure 91 
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The National e-Learning Center was __ Established in 1999 by the Sloan 
established as an independent Foundation, the Online Learning 


entity by the Council of Ministers of | Consortium is a collaborative 


Saudi Arabia, aims to enhancing community of higher education 
trust in the eLearning programs, leaders and innovators, dedicated 
Leading innovation in Learning to advancing quality digital 


digital transformation and enabling teaching and learning experiences 
the integration among educational designed to reach and engage the 

institutions and labor market modern learner - anyone, anytime, 
needs. anyplace. 
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